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The  NATO  Science  and  Technology  Organization 
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Improving  Human  Effectiveness  through  Embedded 

Virtual  Simulation 

(STO-TR-HFM-165) 

Executive  Summary 

Present  and  anticipated  NATO  missions  require  highly  trained  and  capable  military  personnel.  It  is  important 
that  policy,  procedures,  and  technologies  provide  adequate  means  to  prepare  coalition  forces  for  the  full- 
spectrum  of  situations  that  they  are  likely  to  encounter.  An  important  requirement  is  the  need  for  units  to 
deploy  with  little  or  no  notice  and  for  them  to  adapt  effectively  to  evolving  situations.  This  places  military 
units  in  locations  where  they  will  not  have  the  facilities  and  infrastructure  they  had  at  their  home  station 
limiting  their  ability  to  train  for,  and  rehearse  complex  missions. 

The  use  of  Embedded  Virtual  Simulation  (EVS),  within  a  broader  Embedded  Training  (ET)  capability,  is 
seen  as  a  potential  tool  to  provide  more  effective  deployed  training.  EVS  is  a  concept  that  tightly  integrates 
training  and  mission  functionality  into  operational  equipment.  Recent  advances  in  training  concepts,  agent 
technologies,  computers,  communication  and  display  technologies  offer  new  opportunities  for  embedded 
training  and  mission  preparation/rehearsal  capabilities  in  a  highly  mobile  military. 

Deployed  settings  that  provide  little  support  to  users,  e.g.,  lack  of  instructional  staff  and/or  infrastructure 
provide  EVS  its  greatest  challenges.  To  meet  these  challenges  EVS  applications  will  have  to  be  designed 
into  military  systems  with  a  range  of  training  and  training  management  capabilities.  Individual  and  collective 
training  will  require  systems  to  be  equipped  with  display  technology  to  present  detailed  virtual  environments. 
With  further  maturation,  Augmented  Reality  technology  could  be  used  to  present  virtual  targets  and 
maintenance  problems.  Intelligent  agents  would  drive  adversary,  neutral  and  friendly  forces  to  fill  out 
scenarios  and  an  intelligent  tutor  would  provide  feedback  and  training  management  functions.  By  utilizing 
network-enabled  capabilities,  team,  collective,  joint,  and  coalition  training  and  mission  preparation/rehearsal 
(for  warfighting,  peace  keeping  and  maintenance  skills)  are  possible. 

This  report  summarizes  the  findings  of  a  NATO  Research  Task  Group  (RTG)  that  was  established  to  explore 
the  potential  of  embedded  training  with  particular  emphasis  on  EVS.  The  RTG  looked  to  users  for  their 
requirements  and  experiences  with  embedded  training.  The  group  met  with  experts  in  the  component 
technologies:  virtual  environments,  augmented  reality,  intelligent  agents,  intelligent  tutors,  training,  and 
human  interaction  and  performance.  The  contents  and  conclusions  derived  from  these  meetings  and 
discussions  are  summarized  in  the  papers  that  compose  this  volume.  The  RTG  concluded  that  for  successful 
application  of  ET/EVS  to  future  military  systems  user  requirements  (including  characteristics  of  the  user, 
task,  and  environment)  and  training  management  will  have  to  be  considered  early  in  the  design  process. 
Given  the  key  potential  role  of  intelligent  tutor  technology  in  the  success  of  embedded  training,  the  RTG  felt 
that  further  study  should  be  undertaken  to  determine  the  maturity  and  promise  of  tutor  technology.  EVS 
could  become  a  disruptive  technology  by  providing  NATO  forces  a  training  readiness  advantage  on  the 
battlefield,  particularly  if  EVS  systems  can  exploit  the  advances  now  being  made  in  other  fields,  such  as 
secure,  wide  broadband,  wireless  networking  and  intelligence  gathering. 
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Amelioration  de  Pefficacite  humaine  grace  a  la 
simulation  virtuelle  integree 
(STO-TR-HFM-165) 

Synthese 

Les  inissions  actuelles  et  a  venir  de  l’OTAN  necessitent  un  personnel  militaire  hautement  entraine  et 
competent.  11  est  important  que  les  politiques,  procedures  et  technologies  offrent  des  moyens  appropries  pour 
preparer  les  forces  de  coalition  a  l’ensemble  des  situations  auxquelles  elles  peuvent  etre  confrontees.  La 
necessity  de  deployer  des  unites  dans  un  delai  tres  court  ou  sans  delai  et  de  s’adapter  a  des  situations 
changeantes  est  un  facteur  important.  Ces  unites  se  retrouvent  dans  des  lieux  ou  elles  ne  disposent  plus  des 
moyens  ou  infrastructures  necessaires  pour  s’entrainer,  prevoir  et  repeter  de  fagon  optimale  des  missions 
complexes. 

L’utilisation  de  la  simulation  virtuelle  integree  (EVS),  dans  le  cadre  d’une  capacite  d’entrainement  integre 
(ET)  plus  elargie  est  consideree  comme  un  outil  essentiel  pour  permettre  un  entrainement  plus  efficace  en 
projection.  L’EVS  est  un  concept  qui  incorpore  etroitement  des  fonctionnalites  d’entrainement  et  de  mission 
dans  l’equipement  operationnel.  Les  progres  recents  en  matiere  de  concepts  d’ entrainement,  technologies 
d’agents,  ordinateurs,  communications  et  technologies  d’affichage  offrent  de  nouvelles  occasions 
d’integration  des  capacites  d’entrainement  et  des  capacites  de  preparation  /  repetition  dans  un  contexte 
militaire  a  forte  mobilite. 

Les  applications  les  plus  delicates  d’EVS  sont  les  situations  ou  le  site  de  projection  offre  peu  de  soutien  aux 
utilisateurs,  par  exemple,  le  manque  de  personnel  d’instruction  et/ou  d’ infrastructure.  Pour  faire  face  a  ces 
defis,  la  simulation  integree  devra  etre  congue  de  fagon  a  integrer  dans  les  systemes  militaircs  toute  une 
panoplie  de  capacites  d’entrainement  et  de  gestion  des  1’ entrainement.  L’ entrainement  individuel  et  collectif 
requerra  que  les  systemes  soient  equipes  des  nouvelles  technologies  d’affichage  afm  de  presenter  des 
environnements  virtuels  detailles.  Maturation  aidante,  la  technologie  de  Realite  Augmentee  (AR)  pourra 
permettre  la  presentation  de  cibles  virtuelles  et  de  problematiques  d’etrentien.  Des  agents  virtuels  pourraient 
piloter  des  forces  adverses,  amis  ou  neutres  afin  de  creer  des  scenarii  et  des  tuteurs  virtuels  pourraient 
remplir  les  fonctions  d’instructeur.  Les  capacites  reseau-centriques  rendent  possibles  1’ entrainement  en 
equipe,  1’ entrainement  collectif,  interarmees  et  de  coalition,  ainsi  que  la  preparation  /  repetition  de  missions 
(pour  les  actions  de  combats,  d’imposition  et  de  maintien  de  la  paix). 

Le  present  rapport  resume  les  resultats  du  groupe  de  travail  de  l’OTAN  qui  a  ete  cree  en  vue  d’explorer  les 
pistes  potentielles  relatives  a  1’ entrainement  integre,  en  se  concentrant  sur  la  Simulation  Virtuelle  Integree. 
Le  groupe  de  Travail  s’est  rapproche  des  utilisateurs  afin  de  connaitre  leurs  besoins  et  mesurer  leur 
experience.  Le  groupe  a  rencontre  des  experts  des  technologies  associees  :  les  environnements  virtuels,  la 
realite  augmentee,  les  agents  virtuels,  1’ entrainement,  l’ergonomie  et  la  performance  humaine.  Les  reflexions 
et  conclusions  issues  de  ces  reunions  et  discussions  sont  resumees  dans  les  pages  qui  suivent.  Le  groupe  de 
travail  a  conclu  que  la  prise  en  compte  le  plus  en  amont  dans  la  phase  de  conception  des  besoins  des 
utilisateurs  en  matiere  de  EVS/ET  (incluant  les  caracteristiques  de  l’utilisateur,  la  mission  et 
l’environnement)  et  la  gestion  de  l’entrainement  est  un  facteur  de  succes.  Compte  tenu  du  role  potentiel 
important  de  la  technologie  de  tuteur  intelligent  pour  le  succes  de  1’ entrainement  integre,  le  groupe  de  travail 
a  estime  que  des  travaux  doivent  etre  poursuivis  pour  determiner  la  maturite  et  le  potentiel  de  cette 
technologie.  L’EVS  pourrait  se  reveler  une  technologie  de  rupture  en  dotant  les  forces  de  l’OTAN  d’un 
avantage  de  mise  en  condition  operationnelle  sur  le  terrain,  en  particular  si  les  systemes  EVS  peuvent 
exploiter  les  avancees  qui  sont  realisees  dans  d’autres  domaines  comme  les  reseaux  sans  fil  large  bande 
securises  et  le  recueil  d’ information. 
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1.0  BACKGROUND 

Present  and  anticipated  NATO  missions  require  highly  trained  and  capable  military  personnel.  The  military 
have  to  be  well  prepared  to  effectively  and  efficiently  use  state-of-the-art  technology  under  highly  complex 
and  rapidly  evolving  conditions,  e.g.,  ‘three-block-war’  scenarios  (i.e.,  humanitarian  aid,  peace-supporting, 
full  battle  missions).  It  is  important  that  training  policy,  procedures,  and  technologies  provide  adequate 
means  to  prepare  coalition  forces  for  the  full  spectrum  of  situations  they  are  likely  to  encounter. 

A  number  of  factors  are  influencing  training  policies,  procedures  and  technologies.  An  important  factor  is  the 
need  for  units  to  deploy  and  to  adapt  to  evolving  situations.  This  places  them  in  locations  where  they  do  not 
have  the  facilities  and  infrastructure  needed  to  train  for,  plan,  and  rehearse  complex  missions.  HFM-121 
(NATO  RTG-121  Final  Report  (2007))  investigated  Augmented,  Mixed,  and  Virtual  Environments  (AMVE) 
as  a  means  of  providing  an  advanced  and  intuitive  human-system  interaction  for  military  applications.  That 
study  group  identified  military  training  and  education  as  the  main  application  area  for  AMVE  technologies. 
HFM-121  concluded  that  recent  advances  in  training  concepts,  agent  technologies,  computers, 
communication,  and  display  technologies  made  embedding  training  and  mission  rehearsal  capabilities  in 
highly  mobile  military  hardware  both  practical  and  effective.  HFM-121  recommended  that  embedded  virtual 
simulation  technologies  undergo  a  thorough  review  and  analysis  on  conceptual  and  technical  levels.  This 
recommendation  provided  the  impetus  for  the  current  research  technical  group  and  the  contents  of  this 
report. 
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Embedded  Training  (ET)  integrates  training  functionality  into  operational  equipment,  allowing  military 
personnel  to  train  and  rehearse  while  deployed  or  away  from  fixed  training  resources.  A  recent  vision  for  ET 
written  by  the  office  of  the  US  Army’s  Project  Manager  for  Training  Devices  (PM, TRADE,  2011)  stated 
that  ET  minimizes  the  artificialities  of  combat  training  by  maximizing  the  use  of  real  combat  equipment.  The 
vision  statement  stated  that  ET  will: 

•  Enable  training  while  soldiers  are  deployed. 

•  Facilitate  mission  rehearsal  without  the  need  for  special  equipment. 

•  Increase  utility  of  existing  combat  equipment  and  save  money  by  giving  equipment  both  a  combat 
and  a  training  function. 

•  Rapidly  disseminate  combat  experience,  cultural  landscape  awareness,  tactical  adaptations,  and  the 
latest  intelligence  through  training  in  the  war  zone. 

•  Support  home-station  training  by  bringing  training  opportunities  to  locations  that  traditionally  are 
not  equipped  for  training  or  do  not  have  the  space  for  combat  maneuver. 

Embedding  training  capabilities  in  military  equipment  is  not  a  new  concept.  Examples  of  embedded  training 
capabilities  have  been  incorporated  in  defense  computer-based  systems  since  the  late  1950s  (Morrison  & 
Orlansky,  1997).  ET  successes  were  generally  systems  where  space  and  power  requirements  were  not  issues. 
Examples  of  embedded  training  successes  include  Patriot  missile  and  Aegis  Command  and  Control  systems. 
In  their  review  of  embedded  training  utility,  Morrison  and  Orlansky  (1997)  found  56  systems  that 
incorporated,  or  planned  to  incorporate,  embedded  training  capabilities  across  the  US  Army,  Navy,  Air  Force 
and  joint  service  community. 


2.0  DEFINITIONS 
2.1  Embedded  Training 

The  maturity  of  embedded  training  concepts  was  recognized  25  years  ago  when  the  US  Army’s  Vice  Chief 
of  Staff  and  civilian  acquisition  executive  wrote  a  joint  letter  to  the  Army’s  materiel  acquisition  community 
that  established  a  policy  requiring  materiel  developers  to  consider  embedded  training  as  the  preferred 
alternative  for  system  training.  The  policy  also  required  material  developers  to  justify  why  embedded 
training  was  not  viable  before  choosing  other  training  delivery  options  (Thurman  and  Ambrose,  1987). 
During  the  late  1980s,  the  US  Army  Research  Institute  produced  a  ten-volume  guide  on  how  to  implement 
ET  in  Army  systems  (Finley,  Alderman,  Peckham  &  Strassel,  1988).  Among  the  topics  covered  were 
development  of  ET  concepts  (Roth,  1988)  and  requirements  (Roth,  1988),  integrating  ET  into  the  prime 
system  (Evans  &  Cherry,  1988),  incorporating  ET  into  unit  training  (Strasel,  Strasel,  Aldrich,  &  Roth,  1988) 
and  logistics  implications  (Cherry,  Peckham,  Purifoy,  &  Roth,  1988).  The  overview  volume  contained  the 
following  definition  of  embedded  training: 

‘ET  is  defined  as  that  training  which  results  from  features  incorporated  into  the  end  item  of  equipment  to 
provide  training  and  practice  using  that  end  item  equipment.  The  features  may  be  completely  embedded 
within  the  system  configuration  by  software  application  or  a  combination  of  both  software  and  system 
configuration;  or  may  be  executed  by  some  form  of  strap  on  (e.g.,  a  video  disc  player)  or  plug  in  equipment: 
or  a  combination  of  embedded  and  appended  components.  The  feature(s)  MUST  include  stimuli  necessary  to 
support  training:  they  should  include:  performance  assessment  capability,  appropriate  feedback,  and  record 
keeping.  ’  (Finely  et.al.,  1988) 

This  definition  includes  three  types  of  ET :  fully  embedded,  strap-on  and  umbilical.  Fully  embedded  requires 
the  end  item  of  equipment  to  have  been  designed  with  ET  as  part  of  its  system  architecture,  while  strap-on 
and  appended  ET  can  sometimes  be  added  at  a  later  point  in  design.  Embedding  training  in  aircraft  and 
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combat  vehicles  has  been  a  difficult  challenge  due  in  part  to  limitations  on  space,  weight  and  power  in  those 
vehicles.  Recent  miniaturization  of  electronics  and  computer  components  has  reduced  the  need  to  offload 
training  system  components,  making  embedded  training  a  more  viable  option  for  combat  vehicles  and 
aircraft.  However,  modem  systems  that  were  designed  to  incorporate  ET  can  have  a  mixture  of  fully 
embedded,  strap-on  or  appended  components.  An  example  is  the  German  Puma  infantry  carrier.  It  was 
designed  to  have  ET,  and  many  of  the  components  used  for  training  are  also  used  for  fighting  the  vehicle. 
The  image  generator  that  produces  the  visual  virtual  environment  for  training  is  in  effect  an  appended 
component.  The  image  generator  is  loaded  onto  the  rear  of  the  carrier  and  occupies  the  seating  area  normally 
taken  by  an  infantryman  (Modem  Armed  Forces-Puma,  2007).  The  decision  to  include  ET  in  a  military 
system  must  be  made  early  in  the  design  process  to  allow  the  necessary  weight  and  durability  trade-offs  to  be 
made.  Based  on  the  ten-volume  ARI  guide,  Witmer  and  Knerr  (1995)  produced  a  more  usable  guide  to  early 
embedded  training  decisions.  This  guide  went  through  intricate  decision  trees  to  determine  if  ET  was  a 
viable  option  for  a  given  system. 

2.2  Embedded  Virtual  Simulation 

Since  then,  new  technologies  and  findings  have  become  available  to  enable  and  inform  the  use  of  ET.  They 
provide  new  capabilities  for  Embedded  Virtual  Simulation  (EVS),  which  we  define  as  an  enabling 
technology  that  provides  an  interface  to  interactive  simulations  that  reside  within  or  stimulate  operational 
equipment  for  ET.  While  the  enabling  technologies  have  grown  in  number  and  capability,  they  are  also 
potentially  more  cost  effective  than  ever.  In  many  cases,  the  interface  to  the  virtual  simulation  is  the 
operational  equipment’s  own  displays  that  can  be  placed  into  training  mode.  Helmet-mounted  displays  with 
see-through  optics  can  provide  augmented,  mixed  and  fused  reality  that  combine  visual  images  of  the  real 
world,  including  prime  equipment  and  other  people,  both  real  or  synthetic.  Computational  models  of  human 
behavior  can  drive  neutrals’  and  enemy  combatants’  avatars,  as  well  as  substitute  for  missing  teammates. 
Automated  tools  are  being  developed  to  create  training  scenarios,  to  monitor,  record  and  analyze  individual 
and  team/small-unit  performance,  and  to  deliver  timely  feedback  through  an  After  Action  Review  (AAR) 
process.  EVS  can:  (1)  link  local  and/or  distant  trainees  and  instructional  resources;  (2)  provide  a  means  for 
building  individual  or  team  knowledge  and  skills;  and  (3)  be  used  for  requirements  analysis,  test  and 
evaluation,  mission  planning  and  mission  rehearsal. 


3.0  EMBEDDED  TRAINING  APPLICATIONS 
3.1  US  Army’s  Future  Combat  Systems 

The  US  Army‘s  ill-fated  Future  Combat  Systems  (FCS)  program  would  have  been  a  landmark  program  for 
embedded  training.  For  the  first  time  in  a  major  weapons  system  acquisition  program,  training  was  identified 
as  a  Key  Performance  Parameter  at  least  partly  because  embedded  training  and  simulation  was  to  be  the 
primary  training  system  for  FCS.  Since  Key  Performance  Parameters  must  meet  their  key  milestones  in 
order  for  the  system  as  a  whole  to  progress,  embedded  training  would  have  been  a  key  element  in  the  US 
Anny’s  highest  priority  system  of  systems.  FCS  was  to  have  a  virtual  training  system  that  had  much  of  the 
same  functionality  as  the  US  Army's  Close  Combat  Tactical  Trainer.  It  would  also  have  a  training 
management  system,  and  the  capability  to  present  interactive  multimedia  instruction,  as  well  as  to  interact 
with  engagement  simulation  systems  (Shiflett,  2010).  A  training  committee  co-chaired  by  the  Army's 
Program  Executive  Office,  Simulation  Training  and  Instrumentation  and  program  contractor  Science 
Applications  International  Corporation  (SAIC)  jointly  made  key  decisions  regarding  the  design  and 
integration  of  the  trainings  system  with  the  operational  system.  Since  FCS  had  progressed  for  a  number  of 
years,  SAIC  had  developed  a  final  version  of  Brigade  Combat  Team  Embedded  Training  Software  (BETS). 
BETS  consists  of  four  training  products:  Computer-Based  Training;  Virtual  Individual  Operator  Training; 
Leader-Battle  Staff  Training;  and  Live  Training.  While  these  products  will  not  be  used  for  FCS  training, 
BETS  capabilities  can  be  adapted  to  other  systems  and  applications  by  the  addition  of  new  courseware  and 
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simulation  model  data  (SA1C,  2011).  While  the  training  materials  were  designed  for  delivery  in  an 
embedded  training  system,  they  are  also  capable  of  stand-alone  operation.  The  developers  of  BETS  used 
open  standards,  and  the  materials  can  be  used  on  generic  laptops  and  desktops. 

The  potential  for  embedded  training  as  a  means  for  training  battle  command  were  captured  by  Col.  (retired) 
Jim  Shiflett,  the  SA1C  program  manager  for  FCS,  Training,  in  an  unpublished  paper.  Shiflett  (201 1)  pointed 
out  the  advantages  to  integrating  operational  and  training  battle  command  systems  together.  He  argues  that, 
given  the  size  and  complexity  of  battlefield  information  and  sensor  data,  it  will  probably  be  difficult  to 
practice  in  this  type  of  environment  without  employing  simulation,  and  an  embedded  training  capability  is 
likely  the  most  efficient  way  of  doing  this.  Shiflett  identifies  lessons  learned  taken  from  the  FCS  program 
that  should  be  applied  to  future  programs  that  will  likely  employ  an  embedded  training  strategy  for  Battle 
Command.  The  lessons  learned  identified  were: 

•  Training  simulations  can  be  embedded  in  Battle  Command  Systems  if  the  technical  issues  are 
addressed  early  in  the  design  phase  of  the  program.  This  does  not  necessarily  require  lockstep  and 
complex  integration  between  the  simulation  and  Battle  Command  system,  but  could  be  achieved  by 
key  design  principles  and  protocols  that  allow  for  some  decoupling  between  the  Battle  Command 
system  and  the  simulation. 

•  Several  specific  technical  issues,  such  as  time  management  and  simulation  distribution,  need  to  be 
further  addressed  to  find  the  right  range  of  solutions. 

•  With  the  increasing  complexity  of  our  systems  managing  more  varied  and  multi-faceted  sensor  data, 
training  systems  need  to  be  designed  as  part  of  the  Battle  Command  system  to  both  manage 
complexity  of  operation  and  to  support  continuous,  on-demand  training  and  mission  rehearsal. 

3.2  Embedded  Training  in  an  Infantry  Fighting  Vehicle 

The  Puma,  the  new  German  infantry  fighting  vehicle  developed  and  built  by  Kraus-Maffei  Wegmann, 
includes  several  types  of  embedded  training  in  its  design.  In  designing  the  Puma,  it  was  decided  that  a  fully 
embedded  virtual  simulation  system  was  not  feasible  due  to  operational  requirements  placed  on  the  combat 
vehicle’s  infrastructure.  Tactical  driving  training  on  the  Puma  is  achieved  with  minimum  effort  and  only 
requires  simplified  controls.  Advanced  driver  training,  which  would  require  additional  drive-by-wire 
technologies  and  a  motion  base,  was  not  included  since  it  would  be  cost  and  space  intensive.  As  mentioned 
previously,  for  gunnery  and  tactical  training  the  Puma  uses  an  appended  image  generator  which  produces  the 
visual  virtual  enviromnent.  The  image  generator  is  loaded  onto  the  rear  of  the  carrier  and  occupies  the 
seating  area  normally  taken  by  an  infantryman  (Modem  Armed  Forces-Puma,  2007).  The  virtual 
environment  is  visible  through  the  vehicle’s  vision  blocks  and  by  helmet-mounted  displays  that  provide  out- 
of-the-hatch  views.  The  Puma  embedded  training  system  can  function  either  as  a  single  vehicle  or  within  a 
platoon-level  or  higher  network  that  includes  other  vehicles  and/or  simulations  and  an  instructor’s  station. 
Depending  on  the  configuration  of  the  vehicles)  and  the  type  of  training,  additional  equipment  is  added  to  the 
system  (Schmidt,  2010). 

3.3  Embedded  Training  in  Fighter  Aircraft 

Recently,  the  Royal  Netherlands  Air  Force  demonstrated  the  potential  of  EVS  in  fighter  aircraft  (Stokkel, 
2010).  They  embedded  training  capability  into  F-16s  that  enabled  pilots  to  practice  against  virtual  hostile, 
friendly,  and  neutral  players  in  realistic  mission  scenarios.  The  embedded  training  system  injects  virtual  air 
and  surface-to-air  missiles  into  aircraft  mission  systems.  The  virtual  entities  have  simulated  flight  and 
behavioral  characteristics  that  mimic  threats  and  defensive  systems.  The  Netherlands  program  has 
demonstrated  that  fighter  aircraft-based  embedded  training  is  both  technically  feasible  and  effective. 
Decisions  to  implement  embedded  training  into  more  existing  aircraft  and  support  for  embedded  training  in 
the  Joint  Strike  Fighter  program  are  dependent  on  Return  on  Investment  analysis. 
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4.0  MOTIVATION  FOR  RTG-165 

As  the  above  examples  demonstrate,  embedded  training  is  a  concept  that  can  meet  NATO  military  training 
needs,  especially  when  deployments  are  frequent  and  local  requirements  limit  opportunities  to  train  on  each 
element’s  full  range  of  military  tasks  units.  The  programme  of  work  agreed  to  by  RTG-165  was  intended  to 
fully  explore,  from  a  human-centered  design  perspective,  the  conceptual,  functional,  and  technical  promise 
of  EVS  for  ET.  The  RTG’s  work  called  for  a  review  of  activities,  knowledge  and  findings  in  the  area  of 
embedded  virtual  simulation;  analysis  of  implications  of  various  live,  virtual  and  constructive  simulation 
mixes  with  regard  to  training  effectiveness,  human  performance,  and  potential  side-effects;  and  the  provision 
of  recommendations  for  further  research  and  best  practices  to  achieve  embedded  training  for  individual, 
team,  collective,  joint  and  coalition  applications.  The  emphasis  was  to  be  clearly  placed  on  better 
understanding  of  the  training  and  human  performance  issues  that  result  from  the  introduction  of  simulation 
and  other  training  capabilities  to  military  equipment.  The  concern  for  the  human  interface,  training 
approaches,  and  human  performance  and  training  effectiveness  differentiated  the  work  of  this  group  from 
others  that  may  have  reviewed  the  possibilities  for  ET. 

RTG-165  used  several  mechanisms  for  gathering  data  and  opinions  regarding  the  potential  for  EVS.  In 
October  2009,  the  members  of  the  technical  group  organized  a  NATO-sponsored  workshop  (RWS-169).  The 
three-day  workshop  brought  together  experts  who  spoke  about  policy  issues  and  requirements  for  EVS, 
human  effectiveness  in  ET  environments,  human  interaction  in  EVS,  and  learning  in  EVS. 

A  total  of  14  papers  were  presented,  in  addition  to  keynote  addresses  by  four  of  the  technical  group 
members:  Dr.  Lochlan  Magee,  who  defined  EVS,  discussed  the  technologies  involved,  and  provided  context 
for  the  workshop  through  presentation  of  a  possible  scenario  in  which  EVS  was  a  key  element;  Lt.  Col. 
Gerbe  Verhaaf,  who  spoke  about  embedded  training  from  a  military  user’s  perspective;  Dr.  Dirk  Schmidt, 
who  spoke  about  a  researcher’s  perspective;  and  Col.  (Ret.)  James  Shiflett,  who  spoke  about  embedded 
training  development.  The  workshop  combined  these  thought-provoking  presentations  with  time  for  the 
audience  to  have  facilitated  discussion  about  the  EVS  issues  related  to  session  topics.  The  papers  and 
presentations  from  RWS  169  can  be  found  on  the  NATO  Research  and  Technology  Organization  Website, 
www.rto.nato.int  under  ‘Publications’  and  the  title  ‘Eluman  Dimensions  in  Embedded  Virtual  Simulation’. 

The  RTG  developed  a  survey  to  assess  service  members’  awareness  and  use  of,  and  opinions  about,  the  need 
for  embedded  training.  A  total  of  129  respondents  provided  input.  Results  of  the  survey  are  presented  later  in 
this  report.  Finally,  the  RTG  organized  a  session  at  the  2011  International  Training  and  Education 
Conference  (ITEC)  during  which  they  presented  papers  about  the  group’s  findings.  Questions  from  the 
audience  and  a  discussion  followed. 

The  remainder  of  this  report  presents  the  findings  of  RTG-165,  followed  by  recommendations  addressed  to 
NATO  and  allied  Nations  regarding  requirements,  human  interfaces,  technologies,  training  management  and 
mission-support  capabilities,  training  applications  and  other  potential  uses,  and  the  maturity  of  intelligent 
agents  and  their  role  in  EVS. 
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1.0  INTRODUCTION 

1.1  What  are  the  User  Benefits  of  Embedded  Training? 

Extended  periods  of  force  deployment  to  international  theatres  of  war  limits  the  access  of  warfighters  to  the 
resources  required  for  training.  This  particularly  affects  continuation  training,  including  proficiency  training, 
refresher  training,  mission-qualification  training,  and  mission  rehearsal. 

The  possibility  to  perform  ET  exercises  during  deployment,  just  in  time  for  the  warfighters  to  gain  or  refresh 
expertise  for  the  actual  mission  tasks,  would  increase  the  combat  readiness  of  the  force.  The  latter  advantage 
is  not  to  be  underestimated  in  an  era  of  shrinking  defense  budgets  and  an  increasing  Operations  Tempo  (Ops 
Tempo),  with  frequent  out-of-area  deployments  and  reduced  opportunity  for  training  in  the  homeland.  In 
addition,  when  training  in  the  homeland,  ET  would  also  increase  the  efficiency  of  training,  likely  with 
reduced  environmental  impact  (noise  and  emissions),  when  compared  with  traditional  ‘live’  training 
exercises  with  operational  equipment. 

At  the  unit  or  squadron  level,  ET  enables  training  of  the  full  mission  cycle,  on  a  day-to-day  basis,  with  more 
realism  than  with  conventional  training  equipment  or  operational  equipment  without  EVS.  ET  reduces  the 
need  of  mimicking  adversary  assets  with  ‘role  playing’  live  assets,  and  reduces  the  high  costs  for  equipment, 
logistics,  planning  and  personnel  imposed  by  the  use  of  instrumented  ranges. 

1.2  Who  are  the  Operational  Users? 

The  operational  users  addressed  in  the  title  of  this  paper  are  the  individual  warfighters,  who  are  informally 
referred  to  as  ‘trainees’  in  this  paper1.  However,  supervisors,  instructors,  and  other  personnel  involved  in 
training  are  also  considered  as  users  of  ET  and  EVS. 

At  the  level  of  these  end-users,  ET  may  provide  a  high-fidelity  tactical  training  and  rehearsal  environment. 
This  potentially  includes  challenging  training  scenarios,  with  an  optimal  number  of  learning  events,  and 
threat  training  anywhere  and  at  any  time,  which  would  significantly  increase  the  effectiveness  of  training 
hours  for  the  trainees. 


1  While  these  warfighters  may  not  be  trainees  in  the  formal  schoolhouse  sense,  when  they  are  receiving  embedded  training  in 
the  field  they  are  trainees  for  that  short  period. 
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However,  when  considering  EVS  as  a  training  medium,  it  is  legitimate  to  pose  questions  such  as: 

•  Which  tasks  can  be  trained  with  ET  and  how  does  this  training  fit  in  with  other  training  activities  at 
the  level  of  the  individual  operator,  team  and  unit? 

•  What  transfer-of-training  can  be  expected  from  ET  when  the  trainees  are  exposed  to  actual 
battlefield  conditions? 

•  Are  there  elements  in  ET  that  carry  the  risk  of  negative  training? 

•  What  are  the  current  limitations  of  ET? 

1.3  How  to  Define  User  Requirements  for  ET? 

Identification  of  the  need  for  ET,  the  design  and  implementation  of  a  training  program  with  ET,  and  the 
selection  and  specification  of  the  EVS  may  be  supported  by  a  form  of  training  analysis  methodology. 
Various  general  methodologies  have  been  proposed  in  literature,  and  some  of  these  have  been  devised 
specifically  for  simulator-based  training.  An  example  is  the  MASTER  training  design  methodology  (Farmer, 
van  Rooij,  Riemersma,  Joma,  &  Moraal,  1999).  For  the  analysis  of  training  requirements,  such  a 
methodology  typically  takes  into  account: 

•  Mission  characteristics  (mission  systems,  environment,  mission  demands  and  processes). 

•  Task  analysis. 

•  Analysis  of  trainees/target  groups. 

•  Training  objectives. 

However,  since  application  of  EVS  is  different  from  the  use  of  traditional  training  media,  training  analysis 
methods  such  as  MASTER  may  not  be  satisfactorily  applicable  to  ET  requirements.  To  take  full  advantage 
of  ET,  and  related  forms  of  training  such  as  LVC,  may  require  rethinking  of  our  current  approach  to  the 
training  pipeline. 

Also  useful  in  this  context  is  a  publication  by  the  U.S.  Army  (US  Army  Research  Institute,  1996)  that 
provides  guidance  for  ‘early  embedded  training  decisions’.  These  guidelines  are  for  decision  makers  at  both 
operational-system  and  training-system  levels.  One  of  the  obstacles  for  effective  implementation  of  ET  is 
that  multiple  key  decision  makers  are  involved  for  ET.  In  the  traditional  defense  organization,  the  acquisition 
and  exploitation  of  weapon  systems  are  separate  from  those  of  training  systems.  The  different  key  decision¬ 
makers  have  different  responsibilities  and  priorities.  Contrary  to  decision  making  that  deals  with  separate 
training  systems  and  weapon  system,  ET  and  EVS  require  that  the  priorities  of  both  the  operational  and  the 
training  decision  makers  and  their  communities  to  be  reconciled. 

Finally,  since  the  acquisition  of  EVS  integrated  on  operational  equipment  deals  primarily  with  the 
operational  equipment  and  only  secondarily  with  the  EVS,  the  training  requirements  may  be  partly  derived 
through  use  of  a  requirements  analysis  that  is  utilized  in  systems-engineering  management,  (see,  for 
example,  System  Engineering  fundamentals,  U.S.  DoD,  2001,  Chapter  4). 

The  importance  of  defining  user  requirements  is  fundamental  to  the  success  of  ET.  Although  this  may  often 
appear  to  be  stating  the  obvious  to  the  practitioner,  the  user  requirements  are  the  basis  for  a  production  or 
delivery  contract,  and  user  requirements  that  have  not  been  made  explicit  are  unlikely  to  appear  in  the  end 
product. 

In  this  paper,  requirements  are  discussed  in  terms  of  supposed  need,  not  solution.  These  user  requirements 
should  nevertheless  be  achievable,  consistent  with  other  requirements  at  the  user  level.  For  the  purposes  of 
this  paper,  we  make  an  attempt  to  classify  military  user  requirements  under  three  headings:  (1)  user 
interaction  requirements;  (2)  training  management  requirements;  and  (3)  affordability  requirements. 
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2.0  USER  INTERACTION  REQUIREMENTS 

Under  ‘user  interaction  requirements’,  five  classes  of  lower-level  user  requirements  for  ET  are  subsumed. 
These  are: 

•  The  required  level  of  realism  of  the  training  environment,  e.g.,  in  terms  of  visual  cues,  motion  cues, 
aural  cues,  etc. 

•  The  level  of  sophistication  of  other  entities  (i.e.,  other  than  the  human  operators)  that  are  required  to 
constitute  an  effective  training  scenario.  In  most  training  scenarios,  synthetic  entities,  e.g.,  entities 
representing  the  enemy,  must  be  included  in  the  scenario  to  create  meaningful  learning  events.  The 
constituting  models  of,  for  example,  weapons,  behaviours  and  effects,  need  to  have  an  adequate 
level  of  realism. 

•  When  the  ET  is  meant  for  the  training  of  teams,  the  EVS  should  support  communication  and 
coordination  between  team  members. 

•  The  EVS  should  be  user  friendly. 

•  ET  and  interactions  with  the  EVS  should  be  safe  to  use. 

The  authors  devised  this  simple  classification  for  the  current  purposes  only,  based  on  experience  and  without 
attempting  to  provide  a  scientific  justification.  The  fulfilment  of  these  user  interaction  requirements  are 
thought  to  contribute  to  the  trust  the  user  will  have  in  the  EVS.  If  the  training  enviromnent  lacks  an  adequate 
level  of  realism,  despite  the  use  of  operational  equipment,  end-users  will  judge  an  ET  to  compromise  live- 
training  opportunities  and  hence  their  operational  readiness.  The  trust  in  the  EVS  and  the  assessment  that  the 
ET  positively  contributes  to  operational  readiness  will  strongly  determine  the  acceptance  of  ET  by  the  end- 
user  community. 

2.1  Realism  of  the  Perceived  Environment 

When  discussing  the  value  of  any  training,  a  key  issue  is  how  well  the  training  transfers  to  the  tasks  that  the 
warfighter/trainee  must  perform  during  an  actual  mission.  In  this  case,  transfer  of  training  is  defined  as  the 
degree  to  which  trainees  are  able  to  apply  the  competencies  they  gained  in  training  to  the  real  mission.  Much 
has  been  written  on  the  concept  of  transfer  applied  to  simulator-based  training  (see,  for  example,  de 
Fontenilles  etal.,  1996). 

The  main  hypothesis  underlying  the  need  for  realism  in  training  is  that  the  trainee’s  performance  and  the 
processes  involved  in  training  and  transfer  are  largely  linked  to  the  similarity  between  the  tasks  s/he  has  to 
perform  during  training  (i.e.,  task  A)  and  the  tasks  s/he  has  to  perform  in  the  real  mission  (i.e.,  task  B). 

This  implies  that  A  and  B  share  a  set  of  common  characteristics  and  properties.  ET  exploits  this  fundamental 
point  by  using  operational  equipment,  preferably  in  an  environment  that  is  similar  to  the  operational 
environment,  such  that  transfer  from  the  training  tasks  (A)  to  the  operational  tasks  (B)  is  facilitated.  This,  in 
turn,  implies  that  requirements,  with  respect  to  realism,  depend  on  specific  operational  tasks  for  which  the 
training  is  intended.  The  behaviour  to  perform  such  tasks  should  be  expressed  in  learning  objectives,  stating 
the  competence  and  performance  requirements  that  trainees  should  exhibit  at  significant  points  of  the 
training,  e.g.,  after  each  exercise.  This  further  implies  that  an  environment  in  which  a  full  mission  needs  to 
be  mastered  generally  needs  more  realism  than  an  environment  in  which  only  the  control  of  a  piece  of 
equipment  needs  to  be  mastered. 

Requirements  that  relate  to  realism  of  the  perceived  environment  may  be  subdivided  according  to  the 
modalities  in  which  the  environment  is  perceived  by  the  trainee: 

•  Perception  of  self-motion  (via  vestibular,  visual,  tactile,  proprioceptive  and/or  kinaesthetic  senses) 
may  be  the  result  of:  (1)  the  trainee’s  control  of  the  operational  equipment;  or  (2)  external 
disturbances. 
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•  Perception  of  the  visual  world  around  the  trainee  requires  a  specific  image  quality  and  content,  and  a 
certain  legibility  of  visual  cues,  such  as  visual  targets.  Roessingh,  Sijll  and  Johnson  (2003)  analyse 
these  requirements  for  EVS  onboard  fighter  aircraft. 

•  Perception  of  self-motion  and  perception  of  the  visual  world  are  interdependent,  and  when 
simulation  is  involved  in  the  perception  of  both,  specific  design  measures  may  be  needed  to  prevent 
‘simulator  sickness’. 

•  In  reality,  sound  may  be  generated  by  the  operational  equipment  itself  or  may  originate  in  the 
environment.  The  recreation  of  (elements  of)  a  realistic  audible  environment  may  be  required  in  ET. 

The  reader  is  referred  to  Fontenilles  et  al.  (1996),  which  provides  an  extensive  review  of  perception  in 
simulator-based  training  and  related  physical  system  parameters.  However,  the  document  by  Fontenilles  et 
al.  does  not  cover  the  specific  requirements  that  may  be  imposed  on  ET. 

While  legacy  training  simulators  may  be  based  on  application  of  virtual  reality  techniques,  EVS  will  more 
often  rely  on  the  application  of  augmented  reality  techniques  to  provide  a  live  direct  or  an  indirect  view  of 
the  physical,  real-world  environment  whose  elements  are  augmented  by  computer-generated  sensory  input, 
such  as  sound  or  graphics.  Magee,  Sottilare,  and  Roessingh  (201 1)  provide  a  short  paper  with  a  discussion  of 
the  relevance  of  the  aforementioned  factors  for  ET. 

2.2  Realistic  Behaviour  of  Synthetic  Players 

Intelligent  CGFs  or  Non-Player  Characters  (NPCs)  represent  real  people  or  weapon  systems,  including  their 
behaviors  and  cognitive  states  (e.g.,  decision-making  capabilities).  CGFs  are  fully  automated  representations 
of  friends,  adversaries,  or  neutral  ‘characters’.  Just  like  requirements  for  perceived  realism,  user 
requirements  related  to  CGFs  need  to  be  primarily  driven  by  training  objectives. 

Examples  of  user  requirements  may  be: 

•  The  inclusion  of  CGFs  in  EVS  shall  enable  trainees  to  learn  from  opponent  tactics  to  benefit  their 
own  tactical  behaviour. 

•  The  behaviour  of  CGFs  in  a  training  scenario  shall  be  autonomous  and  goal-directed,  using  plans. 

•  Behaviour  of  CGFs  shall  be  affected  by  their  ‘cognitive  state’,  which  takes  into  account,  for 
example,  decision-making  ability,  level  of  expertise,  mental  workload,  situational  awareness,  and 
mental  fatigue. 

The  first  implementations  of  CGFs  for  fighter  aircraft  (for  the  F-16  aircraft,  see  Krijn  &  Wedzinga,  2004, 
and  for  the  F-35  aircraft,  see  Bills  et  al.,  2010)  concerned  two  types  of  CGFs:  adversary  aircraft  and 
adversary  ground  threats  in  the  form  of  Surface-to-Air-Missiles  (SAMs).  The  first  generation  of  adversary 
aircraft  CGFs  is  further  characterized  by  their  lack  of  visual  features.  These  CGFs  will  never  enter  the  visual 
range  of  the  pilot  during  an  EVS  scenario.  Hence,  the  training  application  of  those  adversary  aircraft  NPCs  is 
limited  to  so-called  Beyond  Visual  Range  (BVR)  scenarios.  As  a  result,  their  behaviour  will  be  observable 
only  via  cockpit  instrument  displays  of  onboard  sensors  (radar,  radar  warning  receiver,  possibly  infra-red 
sensors).  The  behaviour  that  these  NPCs  must  demonstrate  needs  to  be  realistic  only  insofar  as  being 
observable  via  those  instrument  displays  in  the  EVS-equipped  aircraft. 

Most  behaviour  of  CGFs  in  EVS  will  be  far  from  trivial.  For  example,  real  enemies  are,  at  least  to  some 
extent,  unpredictable.  They  seek  to  maneuver  themselves  into  a  better  position  as  the  tactical  situation 
changes.  They  react  to  friend  and  foe.  They  are  adaptable.  In  other  words,  they  are  smart. 
The  smart  element  in  the  behaviour  of  these  virtual  opponents  involves  a  number  of  factors.  For  instance, 
they  should  be  able  to  detect  and  identify  targets  to  attack,  but  should  also  be  capable  of  defending 
themselves  against  enemy  action.  To  reach  the  level  of  proficient  behaviour,  intelligent  CGFs  could  be 
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‘trained’,  using  machine  learning  algorithms,  to  instill  the  expertise  that  meets  the  requirements  of  the 
scenarios  and  the  live  participants.  Harrison  et  al.  (2010)  propose  genetic  programming  to  train  NPCs  for 
EVS.  However,  the  maturity  level  of  machine  learning  applications  for  EVS  has  not  been  demonstrated  as 
yet.  Sottilare  and  Roessingh  (2011)  describe  the  application  of  intelligent  agents  in  EVS. 

2.3  Team  Training 

Many  weapons  platforms  are  operated  by  teams,  and  platforms  operate  with  other  platforms  in  many 
missions.  Obviously,  team  training  is  an  important  area  of  application  for  EVS.  Human  interface 
requirements  need  to  allow  coordination  among  team  members  and  platforms.  This  may  require  dedicated 
communication  channels.  The  use  of  EVS  for  team  training  could  promote  unity  in  operational  procedures 
and  doctrines,  and  help  train  effective  communication  techniques.  Training  scenarios  could  be  based,  inter 
alia,  on  actual  battlefield  incidents  involving  factors  related  to  teamwork. 

2.4  Usability 

Usability  concerns  the  ease  of  use  and  leamability  of  the  EVS.  To  determine  user  requirements  for  usability, 
it  should  be  noted  that  there  are  tools  for  usability  methods  (design  and  testing).  Shneiderman  (1980)  and 
Nielsen  (1994)  developed  frameworks  of  system  acceptability,  in  which  usability  was  considered  through  the 
issues  listed  in  Table  2-1: 


Table  2-1:  Usability  Issues. 

Usability  Issue 

Related  test  question 

Leamability 

How  easy  is  it  for  users  (trainees,  supervisors,  instructors)  to  operate  the  EVS 
when  they  use  it  for  the  first  time? 

Efficiency: 

Once  users  have  learned  to  work  with  the  EVS,  how  quickly  can  they  perform  the 
tasks  to  operate  it? 

Memorability 

When  users  return  to  the  EVS  after  a  period  of  not  using  it,  how  easily  can  they 
re-establish  proficiency? 

Errors 

How  many  errors  do  users  of  the  EVS  make,  how  severe  are  these  errors,  and 
how  easily  can  they  recover  from  the  errors? 

Satisfaction 

How  pleasant  is  it  to  operate  the  EVS? 

An  ISO  standard  for  usability  is  1SO/TR  16982:2002:  Ergonomics  of  human-system  interaction — Usability 
methods  supporting  human-centered  design,  which  provides  information  on  human-centred  usability 
methods  that  can  be  used  for  design  and  testing.  Examples  of  usability  requirements  for  EVS  are: 

•  In  deployed  conditions,  (re-)configuration  of  the  operational  equipment  from  ‘training  mode’  to 
‘operational  mode’  or  ‘war  mode’  shall  be  quick  and  easy. 

•  The  logic  behind  operation  of  the  EVS  by  the  trainee  (starting,  stopping,  switching  between  modes, 
switching  between  exercises)  should  be  transparent  for  the  trainees,  with  appropriate  feedback  on 
system  status,  minimizing  the  possibility  for  error. 
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•  The  EVS  should  support  the  use  of  training  data  by  instructors  for  de/briefmg  and  after-action 
review. 

•  If  multiple  EVS-equipped  platforms  are  used  for  team  training,  establishing  communication  links 
and  synchronization  of  the  different  simulations  shall  proceed  automatically. 

2.5  Safety 

ET  is  based  on  the  use  of  operational  equipment.  In  the  ‘training  mode’,  the  trainee  is  supposed  to  act  upon 
this  operational  equipment  as  if  it  were  in  a  real  mission  or  (combat)  operation.  In  some  ET  applications  for 
infantry  fighting  vehicles  or  tanks  with  on-board  EVS,  these  vehicles  would  not  be  able  to  move  when  in 
training  mode,  because  it  would  be  unsafe  to  do  so  (e.g.,  ET  for  the  PUMA  infantry  vehicle,  see  Schmidt, 
2009). 

The  trainee,  who  is  immersed  in  an  embedded  training  scenario,  will  build  Situation  Awareness  (SA)  on  the 
training  situation,  with  augmented  reality.  This  likely  implies  that  the  trainees’  SA  on  the  actual  state  of  the 
equipment  and  its  environment  is  diminished. 

Safety  measures  need  to  be  installed  such  that  the  trainee  can  quickly  recover  its  SA  on  the  actual  situation, 
when  the  situation  requires.  For  example,  in  the  case  of  EVS  on-board  fighter  aircraft,  a  ‘safety  guarding 
system’  should  monitor  that  an  aircraft  remains  in  a  temporary  reserved  airspace  during  the  training.  Such 
system  may  also  provide  a  warning  when  loss  of  separation  with  other  (real)  aircraft  or  terrain  is  imminent. 
In  an  overview  of  the  use  of  embedded  training  for  the  F-35,  Bills,  Flachsbart,  Kern  &  Olsen  (2010) 
emphasized  safety  of  flight  as  a  consideration  within  and  following  the  ‘training’  mode. 

Since  the  radar  display  in  the  cockpit  of  an  EVS-equipped  aircraft  can  contain  both  real  and  virtual 
information  at  the  same  time,  trainees  should  always  be  aware  which  information  is  real  and  which  is  virtual. 
This  helps  them  to  make  the  appropriate  trade-offs  and  decisions  during  the  training  scenario.  It  may  not 
desirable  to  perform  a  potentially  unsafe  maneuvers  or  actions  in  response  to  a  virtual  entity  in  a  training 
scenario,  while  such  maneuver  would  be  totally  justified  in  an  operational  situation.  A  potential 
implementation  for  symbols  on  a  display  is  to  mark  the  virtual  entities  with  a  supplementary  tag.  In  the 
fighter  aircraft  EVS  that  was  used  by  the  Royal  Netherlands  Air  Force  (RNLAF),  this  was  accomplished  by 
attaching  a  small  ‘v’  to  each  virtual  symbol  on  all  displays  where  they  could  appear.  Naturally  such 
information  should  be  designed  carefully  in  order  to  guarantee  positive  transfer  of  training.  Another  effective 
strategy  in  the  design  of  displays  is  to  give  symbols  related  to  real  entities  a  higher  display  priority.  This 
way,  symbols  related  to  virtual  entities  do  never  obscure  those  related  to  real  entities. 

Other  safety  requirements  may  be  associated  with  additional  hardware  that  is  attached  to  the  operational 
equipment.  For  example,  if  a  pilot  helmet  is  modified  to  allow  for  projection  of  virtual  targets  on  the  see- 
through  visor  (augmented  reality),  this  should  not  compromise  pilot  safety  in  terms  of  head  movements,  eye 
sight,  crash  survivability,  high-g  resistance,  ejection  safety,  etc.  For  ET  for  the  F-35,  Bills  et  al.  (2010) 
describe  that  software  for  ET  would  be  partitioned  from  aircraft  flight  software  to  prevent  interference  with 
critical  aircraft  systems.  Virtual  weapon  loads  and  responses  would  be  partitioned  from  onboard  weapon 
control.  Accidental  releases  of  real  bombs  would  be  inhibited  by  simulating  all  weapon  employments 
onboard  the  aircraft  while  in  training  mode.  Actual  weapon  releases  would  be  prohibited  in  training  mode 
and  allowed  only  in  the  ‘live’  mode  with  ET  off. 

Each  implementation  of  EVS  is  likely  to  bear  its  particular  safety  issues.  In  many  cases,  the  development  of 
a  safety  case  or  the  execution  of  a  risk  analysis  (e.g.,  STANAG  4404,  US  DOD.  MIL-STD-882D)  will  be  an 
obligatory  item. 
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3.0  TRAINING  MANAGEMENT  REQUIREMENTS 

Training  management  is  the  continuous  process  at  an  operational  unit,  to  develop,  implement,  deliver, 
evaluate  and  improve  training  programs.  The  goal  of  training  management  is  to  optimize  the  available 
resources,  materials,  guidance,  and  time  to  meet  specific  training  requirements.  Relevant  functions  of 
training  management  for  ET  are  identification  of  training  needs,  design  of  ET  scenarios  and  planning  of  ET 
exercises,  managing  briefing,  after-action  review,  data  collection  and  training  evaluation  (Andrews  and 
Roessingh,  2011). 

3.1  Management  of  Training  Scenarios 

Cannon-Bowers  (2010)  argues  that  ET  is,  in  fact,  scenario-based,  and  the  supposed  training  value  is  based  on 
realistic  scenarios  as  an  effective  means  to  accelerate  the  development  expertise.  Cannon-Bowers 
emphasizes  that  much  of  the  instructional  power  of  ET  is  actually  attributable  to  the  nature  of  the  scenario 
that  is  expressed  in  the  simulated  environment.  Hence  it  is  imperative  to  understand  how  to  design  and 
implement  effective  scenarios  as  a  means  to  optimize  scenario-based  training.  Some  user  requirements 
related  to  the  management  of  training  scenarios  are: 

•  The  EVS  should  be  able  to  run  scenarios  that  address  specific  learning  objectives,  in  accordance 
with  the  full  set  of  learning  objectives  for  ET. 

•  It  should  be  possible  to  script  events  or  ‘triggers’  in  the  scenario  that  allow  the  trainee  to  practice  the 
targeted  learning  objectives  (see,  for  example,  Fowlkes,  Dwyer,  Oser  &  Salas,  1998). 

•  The  EVS  should  support  the  execution  of  newly  created  scenarios.  It  should  be  possible  to  create 
these  scenarios  with  a  commonly  accepted  scenario  management  software  package. 

•  The  EVS  should  support  maintenance,  update  and  modification  of  existing  scenarios. 

3.2  Instruction  and  Feedback 

Feedback  is  an  essential  element  in  ET,  as  it  is  in  all  forms  of  training.  Feedback  must  be  concise  if  it  is  to  be 
useful  in  the  field.  Generally,  the  number  of  feedback  items  should  be  kept  to  a  minimum  if  trainees  and 
supervisors  are  to  make  use  of  them  on  a  regular  basis.  The  number  of  feedback  items  should  be  kept  to  no 
more  than  a  dozen,  less  if  possible  (Andrews  &  Roessingh,  2011).  Different  requirements  for  providing 
training  feedback  to  the  users  of  ET  are  possible,  depending  on  the  specific  application  and  implementation: 

•  The  EVS  may  provide  feedback  to  trainees  and/or  other  personnel  (instructors,  supervisors)  via  the 
original  displays  of  the  operational  equipment.  In  some  applications,  it  may  be  required  to  provide 
the  same  information  via  multiple  on-board  displays  so  that  supervisors  and  trainees  can  view  the 
feedback  at  the  same  time  for  discussion  purposes. 

•  Feedback  information  may  be  carried  from  the  operational  equipment  to  an  off-board  system  (e.g., 
via  a  data  link  or  manually),  such  that  the  supervisor  can  review  performance  for  an  entire  team  with 
the  team  that  has  trained  on  multiple  weapon  systems  (a  tank  platoon,  multiple  aircraft  involved  in 
an  air  campaign).  Alternatively,  the  team  can  review  their  performance  without  a  supervisor  present. 

•  Feedback  may  be  provided  to  the  trainee  without  involvement  of  an  instructor  or  supervisor,  i.e., 
fully  automatic.  In  such  case,  the  EVS  may  merely  provide  the  instructor  or  supervisor  a  summation 
of  performance.  Alternatively,  the  EVS  can  be  designed  such  that  all  feedback  will  be  provided  to 
the  instructor  or  supervisor  and  he/she  provides  the  feedback  to  the  trainee.  Generally,  feedback  via 
an  instructor  or  supervisor  may  be  more  desirable  in  initial  stages  of  training,  while  instructor 
involvement  may  not  be  required  for  mission  rehearsal  under  deployed  conditions. 

•  Feedback  should  be  provided  in  a  timely  fashion,  as  this  can  greatly  aid  the  training  process. 
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•  This  doesn’t  mean  that  all  feedback  information  should  be  provided  as  quickly  as  possible. 
Provision  of  feedback  for  higher  order  cognitive  skills  like  complex  decision  making  is  preferably 
delayed  to  facilitate  learning  retention. 

3.3  Performance  Evaluation 

An  important  requirement  for  training  management  in  EVS  should  be  the  precise  detection  and  registration 
of  the  trainees’  level  of  expertise  such  that  subsequent  ET  sessions  and  scenarios  can  be  adapted  to  the 
appropriate  level,  yielding  the  highest  training  value.  This  also  highlights  the  requirement  that  trainee 
performance  during  ET  exercises  should  be  carefully  recorded  for  the  purpose  of  selecting  subsequent 
training  activities. 

3.4  Management  of  Training  Sessions 

ET  may  diminish  the  role  of  the  human  instructor  because  the  operation  equipment  that  hosts  the  EVS  most 
often  doesn’t  provide  physical  space  for  a  human  instructor.  An  obvious  way  forward  here,  is  the 
replacement  of  the  instructor  by  an  intelligent  tutoring  system  (see  e.g.,  Sottilare,  2010,  Jensen,  Mosley, 
Sanders  &  Sims,  2010). 

At  a  meta-instruction  level,  the  training  sessions  could  be  managed  by  a  Learning  Management  System 
(LMS)  which  (1)  keeps  record  of  exposure  to  specific  training  events  (counting),  (2)  schedules  trainee  events 
in  order  to  optimize  the  learning  process,  (3)  performs  scenario  configuration  management  (which  scenarios 
are  updated  and  ready  for  use),  and  (4)  performs  continuing  trainee  profiling  on  the  basis  of  performance 
measures.  A  possible  user  requirement  is  to  provide  a  coupling  between  the  EVS  and  these  (off-board)  LMS 
functions. 

4.0  AFFORDABILITY 

The  decision  to  introduce  ET  in  the  military  organization  will  be  largely  based  on  projected  cost  savings 
while  combat  effectiveness  need  to  be  maintained  or  increased,  in  other  words,  making  training  more 
affordable,  ft  will  routinely  be  required  during  an  acquisition  process  to  provide: 

•  an  estimate  of  the  value  of  the  added  capability  of  ET  (in  terms  of  an  increased  OPS-level). 

•  an  estimate  of  the  costs  of  ET. 

•  an  estimate  of  the  cost  savings  of  ET  relative  to  current  training  practice. 

•  a  cost  comparison  between  alternative  training  methods. 

The  aforementioned  guide  for  early  embedded  training  decisions  by  the  US  Army  Research  Institute  (1996) 
provides  a  helpful  worksheet  to  compare  ET  with:  (1)  training  using  just  the  operational  equipment  without 
EVS,  and  (2)  training  using  stand-alone  training  devices.  Four  different  cost  categories  are  distinguished: 

•  Design  and  development  costs. 

•  Procurement  costs. 

•  Maintenance  costs. 

•  Operations  costs. 

A  thorough  source  for  quantitative  modeling  of  training  costs  is  provided  by  Orlansky  (1989),  which 
provides  useful  indicators  for  training  and  cost  effectiveness,  including  the  Transfer  Effectiveness  Ratio 
(TER),  the  Training  Cost  Ratio  (TCR),  the  Cost  Effectiveness  Ratio  (CER)  and  the  Simulator  Utilization 
Ratio  (SUR).  If  there  are  data  on,  or  estimates  of,  cost  and  effectiveness  for  two  methods  of  training,  the 
decision  diagram  (Table  2-2)  can  be  used  to  choose  between  them  (Orlansky,  1989). 
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Table  2-2:  Decision  Diagram  between  Two  Methods  of  Training. 


LESS  EFFECTIVE 

SAME 

MORE  EFFECTIVE 

LESS  COST 

Uncertain 

Adopt 

Adopt 

SAME  COST 

Reject 

Uncertain 

Adopt 

MORE  COST 

Reject 

Reject 

Uncertain 

More  recently,  cost-utility  analysis  (CUA),  a  form  of  economic  analysis  used  to  guide  procurement 
decisions,  has  been  introduced  to  evaluate  the  utility  of  training  (e.g.,  Arthur  et  al,  201 1). 

A  practical  example  of  projected  training  savings  with  implementation  of  ET  for  the  F-35  Joint  Strike 
Fighter  is  shown  in  Table  2-3.  ET  is  planned  to  be  introduced  in  a  specific  training  syllabus  (see  Bills  et  al., 
2010): 

‘In  terms  of  student  costs,  the  projection  reduction  is  14  Red  Air  sorties  [i.e.,  sorties  with  fighter  aircraft  that 
mimic  the  enemy’s  tactics],  equivalent  to  23.8  flight  hours  for  $100k,  and  Surface  to  Air  Missiles/Electronic 
Warfare  range  reductions  of  7.2  hours  for  $25. 5k.  Looking  at  start  up,  calendar  years  2015  through  2017, 
the  projected  cost  avoidance  is  $411.5M.  When  the  F-35  reaches  steady  state  at  calendar  year  2030,  the 
projected  cost  avoidance  is  $1,046M.  Over  the  expected  life  cycle  of  calendar  years  2013  through  2057,  the 
program  savings  could  reach  $2,976M.  ’ 

Table  2-3:  F-35  Embedded  Training  Cost  Avoidance  Implementing  Activity  Model  (taken  from 

Bills  et  al.,  2010). 


($M) 

Air  to 

Air 

Air  to 
Ground 

Total 

Start  Up 

$67.6 

$21.6 

$89.2 

CY16 

$103.3 

$30.8 

$134.1 

CY17 

$146.5 

$41.7 

$188.2 

Steady  State  |K£lI|f^| 

$775.5 

$270.5 

$1,046.0 

Life  Cycle 

$2,370.0 

$606.0 

$2,976.0 

It  is  important  to  balance  all  advantages  and  disadvantages  of  ET  in  a  cost  analysis.  By  properly  defining 
requirements,  advantages  of  ET  can  be  exploited  to  the  maximum  while  disadvantages  can  be  minimized  or 
avoided.  Several  examples  of  advantages  are  listed  below.  (Finly,  Alderman,  Peckham  &  Strasel,  1988;  Bills 
et  al,  2010;  Roessingh  &  Verhaaf,  2010): 

•  The  capability  to  provide  training  (e.g.,  refresher,  sustainment,  proficiency,  continuation  and 
mission  qualification  training)  that  is  resident  in  the  operational  unit,  also  during  extended  periods  of 
weapon  platform  deployment  to  international  theaters  of  war. 

•  ET  is  fielded  and  maintained  concurrent  with  the  operational  equipment. 
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•  There  is  a  reduced  need  for  other  training  equipment  (simulators,  trainers)  with  potential  cost 
savings  in  procurement,  maintenance,  infrastructure  and  operation  of  this  equipment. 

•  ET  reduces  wear  and  tear  on  operational  equipment,  thereby  decreasing  maintenance  costs  and 
manpower  requirements,  e.g.,  in  the  case  of  the  F-35,  ET  eliminates  the  need  to  carry  actual 
weapons  during  a  training  mission. 

•  ET  reduces  the  number  of  ‘live  assets’  required  for  role-playing  (e.g.,  mock  enemies)  by  using 
constructive  entities. 

•  There  is  reduced  need  to  use  large  instrumented  training  ranges.  For  example,  highly  realistic  day- 
to-day  fighter  pilot  training  for  RNLAF  pilots  could  take  place  in  reserved  airspace  above  the  North 
Sea.  Through  the  use  of  virtual  enemy  aircraft  the  required  airspace  for  air-air  engagements  is 
further  reduced.  Apart  from  logistic  advantages,  this  results  in  abatement  of  noise  and  emission. 

•  There  are  various  options  to  reduce  the  training  management  burden  at  the  operational  unit  (for 
example,  the  EVS  provides  training  capability  when  an  instructor  is  not  present  or  readily  available, 
see  Section  3.0). 

•  Training  is  standardized  across  operational  units. 

•  As  a  by-product,  high  quality  job-aids  for  the  system  could  be  developed  easily  from  the  ET 
materials. 

Also,  potential  disadvantages  must  be  taken  into  account,  for  example: 

•  Costs  associated  with  the  operational  equipment  will  increase  with  EVS. 

•  The  operational  equipment  is  required  in  order  to  conduct  training2.  Additional  systems  may  be 
required  solely  for  the  purposes  of  providing  training. 

•  Components  of  the  EVS  may  need  to  be  hardened  or  ruggedized,  and  therefore  may  be  more 
expensive  than  they  would  need  to  be  when  used  in  conventional  training. 

•  ET  may  cause  additional  wear  and  tear  on  components  of  the  operational  equipment. 

•  ET  may  not  be  available  in  the  initial  phase  of  deployment/mobilization,  particularly  if  personnel 
and  equipment  are  transported  separately. 

•  EVS  components  may  take  up  space  and  add  weight  to  the  operational  equipment. 

Bills  et  al.  (2010)  further  argue  that  current  experience  with  and  research  into  ET  is  still  too  sparse  for  a 
conclusive  analysis  for  training  effectiveness  and  efficiency. 


5.0  CONCLUSIONS  AND  RECOMMENDATIONS 

This  paper  provided  an  overview  of  possible  user  requirements  for  ET.  A  thorough  analysis  of  the 
operational  user  requirements  is  needed  to  realize  the  numerous  advantages  of  ET.  This  includes  training 
needs  analysis  providing  a  set  of  training  objectives  for  the  ET,  requirements  analysis  and  cost  analysis. 
Operational  user  requirements  have  been  classified  in  three  categories:  user  interaction,  training 
management,  and  affordability. 


2 

“  However,  ET  may  also  be  integrated  in  training  equipment,  e.g.,  training  aircraft.  In  the  EUCLID  (European  Collaboration  for 
Long  term  in  Defence)  RTP  11.12  program,  ET  was  integrated  in  the  Aermacchi  MB  339  training  aircraft  in  2003. 
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User  Interaction 

User  interaction  requirements  include  realism  of  the  perceived  environment,  realism  of  (intelligent)  CGFs, 
team  training  requirements,  usability  requirements  and  safety  requirements. 

The  required  realism  in  terms  of  perceived  self-motion,  the  visual  and  audible  environment,  etc.  will  strongly 
depend  on  the  training  objectives.  Some  training  objectives  may  require  solutions  that  are  not  yet  available  in 
current  EVS  designs.  Augmented  reality  is  a  key  technology.  Conflicts  between  visual  cues  and/or  vehicle 
movement  must  be  avoided  to  prevent  simulator  sickness. 

The  realistic  behaviour  of  synthetic  players  or  CGFs  will  often  be  a  strong  user  requirement.  Required 
flexibility  in  scenarios  will  require,  for  example,  the  behaviour  of  enemies  to  be  ‘smart’,  i.e.,  intelligent  and 
autonomous.  For  team  training,  coordination  and  communication  among  team  members  and  platforms  will 
be  required,  possibly  using  dedicated  data  links.  Usability  aspects  of  the  EVS  concern  operation  by  trainees 
and  instructors  and  re -configuration  of  the  platform.  Main  safety  concern  is  the  recovery  of  situation 
awareness  from  the  training  situation  to  ‘real  life’,  for  example  in  case  of  emergencies  or  unusual  situations. 

Training  Management 

The  position  of  ET  in  the  training  program,  like  any  other  type  of  training,  should  optimize  the  learning 
curve  towards  the  required  mission  competencies.  This  requires  an  analysis  of  training  objectives  to  which  it 
could  contribute,  given  the  full  range  of  operational  tasks.  The  notion  that  ET  is  ‘scenario-based  training’ 
requires  that  an  instructor  or  scenario  manager  can  create,  update  and  modify  meaningful  scenarios.  Further 
training  management  requirements  relate  to  feedback,  performance  evaluation  and  the  diminished  role  of  the 
instructor. 

Affordability 

ET  should  result  in  cost  savings  while  combat  effectiveness  need  to  be  maintained  or  increased.  It  is 
important  to  balance  all  advantages  and  disadvantages  of  ET  in  a  cost  analysis.  The  lessons  learned  from  the 
last  two  decades  are  that  those  responsible  for  Defence  budget  cuts  have  a  tendency  to  choose  for  combat 
capabilities  at  the  expense  of  training.  Although  the  consideration  of  training  as  a  closing  entry  at  the  bottom 
of  the  list  is  comprehensible,  it  doesn’t  seem  logical.  A  very  substantial  part  of  combat-effectiveness  is 
attributable  to  the  quality  of  the  training  and  the  combat  readiness  of  the  warfighter. 

Recommendations  for  Research  &  Development 

There  is  generally  a  lack  of  research  results  on  the  training  effectiveness  and  efficiency  of  ET.  The  critical 
components  of  the  EVS  and  the  training  design  that  is  most  suitable  for  ET  are  largely  unknown.  A  training 
design  methodology  that  is  used  for  the  development  of  training  should  explicitly  consider  ET  and  EVS  as 
options.  In  current  ET  applications,  training  management  methods  and  techniques  often  seem  to  be 
neglected. 

Important  techniques  that  are  likely  to  be  required  in  many  ET  applications  are:  (1)  visualization  of  virtual 
entities  in  augmented  reality;  (2)  intelligent  CGFs;  (3)  intelligent  tutoring;  and  (4)  data  links  for  ET  (e.g.,  for 
team  communication  and  team  coordination  in  case  of  team  training,  scenario  manipulation,  feedback, 
interventions  by  instructors,  etc.). 

Other  areas  for  R&D  are:  cost  models  for  ET,  safety  and  regulatory  aspects  of  ET,  and  certification  of  EVS 
equipment. 
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1.0  INTRODUCTION 

The  members  of  the  NATO  Human  Factors  and  Medicine  Panel  -  Research  Task  Group  (NATO  HFM- 
RTG-165)  developed  a  survey  to  learn  more  about  the  military  population  that  have  used  and  might  be  using 
Embedded  Virtual  Simulation  (EVS)  technologies  in  the  future.  Comprised  of  twenty  questions,  the  survey 
also  served  to  ascertain  the  level  of  awareness  of  EVS  in  the  military  population  and  their  opinions  of  current 
embedded  training  capabilities  in  theatres  of  war.  The  questions  included  in  the  survey  are  attached  as  an 
annex  to  this  chapter.  This  chapter  summarizes  the  survey  data  to  provide  insight  into  the  potential  of  EVS. 
Recommendations  are  also  provided  for  the  practical  implementation  and  deployment  of  future  EVS  based 
on  the  survey  data. 

EVS  is  anticipated  to  be  deployed  as  a  tool  to  support  future  embedded  training  in  theatres  of  war.  EVS  is 
considered  the  enabling  technology  and  embedded  training  is  the  capability  available  to  military  users.  Since 
our  focus  is  on  user  opinions  and  their  view  is  less  on  the  technology  (EVS)  and  more  on  the  capability 
(embedded  training),  we  have  opted  to  use  the  term  ‘embedded  training’  when  referring  to  user  opinions  and 
EVS  when  discussing  the  enabling  technology. 

Embedded  training  concepts  have  been  around  for  over  30  years,  but  have  not  grown  to  meet  those  early 
expectations  to  support  ‘training  on  the  move’  or  ‘training  in  deployed  locations’.  The  motivation  for  the 
survey  described  in  this  chapter  was  to  learn  what  users  think  about  embedded  training,  their  level  of 
exposure  to  EVS  technologies,  and  current  use  in  theaters  of  war  today. 

To  gather  information  about  the  users’  exposure  to  and  opinions  on  embedded  training,  the  Panel  developed 
a  survey  for  anonymous  input  by  the  military  members  through  ‘Survey  Monkey’  where  data  was  collected 
and  analyzed.  The  questions  detailed  in  the  survey  are  attached  as  an  appendix.  This  chapter  serves  to 
describe  the  survey  participants  (potential  embedded  training  users),  their  knowledge  and  use  of  training  in 
theater,  and  their  opinions  about  embedded  training  in  general. 


2.0  USER  PROFILE  SURVEY  RESULTS 

The  intended  population  for  this  survey  was  a  NATO  military  population  with  a  normalized  distribution 
across  various  ranks  (enlisted  and  officer),  services  (e.g.,  Army,  Navy  and  Air  Force)  and  operational 
occupational  series.  The  actual  sample  distribution  (N  =  129  participants)  was  much  different  from 
expectations,  with  a  high  concentration  of  Army  participation  (93  percent),  low  participation  from  Air  Force 
personnel  (6  percent)  and  no  participation  from  Naval  (0  percent)  personnel,  including  Marines.  It  is 
expected  that  additional  participation  from  naval  personnel  would  positively  influence  the  statistics  described 
in  this  chapter. 


STO-TR-HFM-165 


3-1 


RESULTS  OF  THE  EMBEDDED  VIRTUAL  SIMULATION  USER  SURVEY 


organization 


There  was  some  international  sensitivity  about  collecting  data  about  the  participants’  countries  and  their 
ranks,  but  the  survey  did  classify  participants  as  Enlisted  (77  percent),  Officer  (19  percent)  and  unknown  (4 
percent).  Their  occupational  series  varied  from  combat  to  combat  services  to  combat  series  support. 
Participants  classified  their  competency  levels  in  their  occupational  specialties  as  novice  (13  percent), 
journeyman  (33  percent)  or  expert  (54  percent),  demonstrating  a  high  level  of  training/experience  across  the 
survey  sample.  Training  was  reported  by  74  percent  of  the  participants  as  an  important  part  of  their  pre¬ 
deployment  preparation.  As  noted  in  Figure  3-1,  over  73  percent  of  the  participants  noted  that  they  had  been 
deployed  to  a  theater  of  war  within  the  last  two  years.  This  lends  credibility  to  their  opinions  regarding  the 
frequency  and  effectiveness  of  training  in  theater  and  the  potential  need  for  embedded  training  options. 


□within  the  last  2  years 
■2^4  years  ago 

□  4-6  years  ago 

□  6-8  years  ago 

■  8-10  years  ago 

■  more  than  1 0  ye  ars  ago 


Figure  3-1 :  Last  Deployment  to  a  Theater  of  War. 


When  asked  about  how  effectively  they  were  prepared  to  carry  out  their  specialty  duties  when  they  arrived  in 
theater,  participants  reported  they  were  well  prepared  (mean  =  6.36  on  a  7  point  Likert  scale  where  7  was 
defined  as  fully  prepared).  The  distribution  of  preparation  effectiveness  scores  is  shown  in  Figure  3-2.  This 
distribution  shows  89.8  percent  of  the  participants  felt  they  were  at  least  moderately  prepared  and  over 
3 1  percent  felt  they  were  fully  prepared  to  execute  their  duties. 


poorly  moderately  fully 

prepared  prepared  prepared 


reported  preparation  level  at  start  of  deployment 

Figure  3-2:  Preparation  Level  of  Participants. 
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When  asked  about  their  frequency  of  training  in  theater,  72.4  percent  reported  that  they  trained  daily  in  their 
specialty  areas,  and  an  additional  12.6  percent  indicated  that  they  trained  at  least  weekly.  Over  83  percent  of 
the  participants  indicated  that  their  frequency  of  training  while  deployed  was  adequate  to  maintain 
competency  in  their  occupational  specialties.  The  combination  of  reported  preparation  and  frequency  of 
training  in  theater  indicate  participants  were  ready  to  do  their  assigned  duties,  and  continued  to  maintain  a 
high  level  of  proficiency  through  frequent  training.  The  survey  data  does  not  indicate  the  costs  of  deploying 
this  training  capability,  nor  does  it  indicate  its  level  of  effectiveness. 

When  asked  about  the  type  of  training  in  which  they  participated,  6 1  percent  of  the  respondents  participated 
in  live  training  only,  6  percent  participated  in  computer-based/simulator-based  training  only,  and  31  percent 
reported  that  they  participated  in  both  live  and  computer-based/simulator-based  training  exercises.  This  may 
be  indicative  of  low  availability  of  computers/simulators  for  training  in  theater.  If  this  is  true,  consideration 
should  be  given  to  enhance  the  availability  of  computer-based/simulator-based  training  in  theater  to  maintain 
skills  and  to  train  for  missions  that  might  otherwise  not  be  practical/safe  in  a  live  training  environment. 
Embedded  training  solutions  are  a  viable  alternative  to  deploying  a  separate  simulator  infrastructure. 


3.0  EMBEDDED  TRAINING  SURVEY  RESULTS 

The  participants  were  asked  if  they  had  any  type  of  embedded  training  capability  available  while  deployed; 
77  percent  responded  ‘no’  and  only  23  percent  responded  ‘yes’.  When  the  23  percent  who  reported  that 
embedded  training  was  available  were  asked  if  they  used  the  embedded  training,  59.5  percent  of  the 
23percent  said  ‘yes’  and  40.5  percent  said  ‘no’.  This  group  of  embedded  training  users  translates  to  only 
approximately  14  percent  of  the  total  survey  participants. 

In  the  embedded  training  user  group,  the  type  of  embedded  training  available  used  in  theater  included 
individual  only  (21  percent),  collective  only  (58  percent)  and  both  (21  percent).  The  participants  rated  the 
usefulness  of  the  embedded  training  they  used  in  theater  as  highly  useful  (mean  =  6.32  on  a  7  point  Likert 
scale  where  7  was  defined  as  very  useful).  The  distribution  of  usefulness  scores  is  shown  in  Figure  3-3.  The 
data  shows  that  over  86  percent  of  the  participants  rated  the  embedded  training  they  used  as  at  least 
moderately  useful.  Given  the  experience  level  and  recent  deployments  of  the  participants,  the  usefulness  data 
indicates  that  embedded  training  solutions  would  be  a  welcome  addition  to  the  training  regimen  in  theaters  of 
war. 


is 


not  use  fill  mo  deratelv  very  use  fill 

useful 


reported  usefulness  of  embedded  training  used  in  theater 
Figure  3-3:  Usefulness  of  Embedded  Training  in  Theaters  of  War. 
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4.0  DISCUSSION 

The  most  glaring  facts  uncovered  during  this  study  were  the  current  lack  of  embedded  training  solutions  for 
use  in  theater,  and  the  high  level  of  understanding  of  what  embedded  training  was  and  how  it  could  help. 
Only  23  percent  of  survey  participants  had  access  to  embedded  training  capabilities  during  deployment,  yet 
the  same  group  was  able  to  provide  relevant  recommendations  about  the  relevant  features  they  expected  to 
see  in  an  embedded  training  solution.  Most  respondents  noted  that  a  realistic  environment  was  important  in 
simulating  the  stress  of  the  operational  environment,  and  many  included  automation  as  an  important  feature 
of  an  embedded  training  environment.  Automation  included  capabilities  that  respondents  described  as 
automated  data  collection,  intelligent  tutoring  systems,  automated  After  Action  Review  systems  (AARs),  and 
virtual  non-player  characters  to  represent  the  roles  of  non-trainee  participants. 

Even  those  with  little  or  no  exposure  to  embedded  training  shared  positive  thoughts  on  how  embedded 
training  might  be  used.  Some  quotes  are  included  below: 

•  ‘If  all  units  were  equipped  with  embedded  training,  it  might  make  it  easier  for  those  who  are 
‘disconnected’  due  to  location,  to  stay  current,  or  get  more  up  to  date  information/training  while 
deployed  in  remote  areas.’ 

•  ‘Although  I  have  no  experience  with  it,  I  think  embedded  virtual  training  is  an  excellent  concept  and 
very  useful  for  increasing  experience  in  combat  related  situations.’ 

•  ‘It  is  much  better  to  use  pre-deployment  training  -  you  MUST  arrive  ready  for  war,  but  In-theater 
training  (embedded)  could  be  used  for  currency,  trying  new  tactics,  etc.’ 

•  ‘Although  my  experience  is  a  bit  dated,  embedded  training  would  allow  more  realistic  scenarios.’ 

•  ‘Stress  is  the  key.  Simulators  can  be  used  for  training  tactics,  but  proximity  to  another  aircraft,  actual 
shaking  and  immense  ground  rush  cannot  be  simulated.  You  only  get  that  in  the  aircraft.’ 


organization 


5.0  CONCLUSIONS  AND  RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

As  noted  in  the  introduction,  embedded  training  concepts  have  been  in  use  for  over  30  years,  but  are  not  the 
ubiquitous  training  solution  envisioned.  The  low  percentage  of  embedded  training  users  (14  percent)  in  our 
survey  results  can  be  attributed  to  the  low  availability  of  embedded  training  systems  and  direct  competition 
with  existing  training  programs  in  theater.  Based  on  the  survey  results,  it  was  obvious  that  exposure  to 
embedded  training  concepts  was  low. 

Still,  our  survey  hints  at  the  utility  of  embedded  training.  Those  in  the  survey  who  used  embedded  training 
were  overwhelmingly  positive  with  over  86  percent  of  users  rating  their  training  at  least  moderately  useful 
and  over  43  percent  rating  their  training  very  useful. 

The  practicality  and  utility  of  embedded  training  concepts  have  yet  to  be  exploited  on  a  large  scale.  It  will  be 
important  moving  forward  to  share  the  results  of  this  survey  and  to  provide  success  stories  on  the 
deployment  of  embedded  training  solutions  to  educate  users,  developers  and  policy  makers  about  the 
usefulness  of  embedded  training  capabilities. 
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5.2  APPENDIX  A 

Embedded  virtual  simulation  survey 

1)  Are  you  now  or  have  you  been  a  member  of  the:  Army,  Navy,  Air  Force,  Marine  Corps, 
Other? 

2)  Have  you  been  deployed  to  Afghanistan  or  similar  areas: 

a)  within  the  last  2  years 

b)  2-4  years  ago 

c)  4-6  years  ago 

d)  6-8  years  ago 

e)  8-10  years  ago 

f)  more  than  1 0  years  ago 

3)  What  was  your  specialty  or  your  job  duties  while  deployed? 

4)  While  deployed,  did  you  consider  yourself  (select  one): 

a)  a  novice  in  your  specialty? 

b)  a  journeyman  in  your  specialty? 

c)  an  expert  in  your  specialty? 

5)  How  did  you  prepare  for  your  individual  duties  within  your  specialty  before  you  were 
deployed? 

6)  How  did  your  unit/ collective/team  prepare  for  your  missions  before  you  were  deployed? 

7)  How  well  prepared  for  your  duties  do  you  feel  you  were  when  you  arrived  in  theatre? 

8)  In  theatre,  how  often  did  you  perform  tasks  relevant  to  the  specialty  you  trained  for? 

9)  Were  you  able  to  do  any  training  while  in  theatre? 

10)  What  did  the  training  consist  of?  Please  elaborate  (e.g.,  computer-based,  simulator-based,  live 
training). 

11)  How  frequently  were  you  able  to  train  in  theater?  Daily?  Weekly?  Monthly?  Hardly  ever? 

12)  Did  you  feel  the  frequency  of  training  was  adequate? 

13)  Did  you  feel  the  quality  of  the  training  was  adequate? 

14)  Did  your  military  equipment  have  any  type  of  embedded  training  capability? 

1 5)  Did  you  use  this  embedded  training  capability  in  theater? 
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16)  On  a  scale  of  1  (not  useful)  to  7  (very  useful),  please  rate  the  usefulness  of  the  embedded 
training  capability  you  used  in  theater. 

17)  Do  you  think  that  an  embedded  training  capability  is  needed  to  train  your  specialty  in  theater? 

1 8)  In  your  opinion,  which  capabilities/features  need  to  be  included  in  embedded  training  systems 
to  be  worthwhile  as  training  tools?  (e.g.,  realistic  environment,  simulation  of  stress,  intelligent 
tutors,  automated  after  action  review,  mission  editor.) 

19)  If  you  had  embedded  training  in  theatre,  how  do  you  think  it  should  be  used?  For  example, 
should  it  be  used  just  before  doing  an  actual  mission,  for  refresher  training,  individual 
training,  collective  training,  or  other? 

20)  Are  there  any  other  thoughts  on  the  topic  you  would  care  to  share  with  us? 


3-6 


STO-TR-HFM-165 


organization 


Chapter  4  -  TRAINING  MANAGEMENT  IN  EMBEDDED  VIRTUAL 

SIMULATIONS 


Dee  H.  Andrews,  Ph.D. 

Air  Force  Research  Laboratory  -  71 1th  Human  Performance  Wing 
UNITED  STATES 

Dee.Andrews@mesa.afmc.af.mil 

Jan  Joris  Roessingh,  Ph.D. 

Netherlands  National  Aerospace  Laboratory 
THE  NETHERLANDS 

Jan.Joris.Roessingh@nlr.nl 


1.0  INTRODUCTION 

1.1  The  Relevance  of  Training  Management  for  EVS 

Training  management  is  the  continuous  process  in  an  organization,  e.g.,  an  operational  unit,  to  develop, 
implement,  deliver,  evaluate  and  improve  training  programs.  The  goal  of  training  management  is  to  optimize 
the  available  resources,  materials,  guidance,  and  time  to  meet  specific  training  requirements.  Relevant 
functions  of  training  management  for  EVS  are  for  example: 

•  Defining  Training  Needs:  define  specific  competency  needs,  in  first  instance  for  trainees,  but  also 
for  related  personnel  such  as  senior  personnel  overlooking  the  EVS  exercises  and  staff  that  prepares 
the  EVS  scenarios,  identify  training  exercises  which  address  these  needs. 

•  Designing  and  planning  of  EVS  exercises,  including  facilitating  of  mission  planning,  careful 
consideration  of  training  objectives  and  instructional  strategies,  planning  of  briefing  and  debriefing, 
planning  of  data  collection. 

•  Training  Implementation,  including  management  of  EVS  related  materials,  scenarios, 
instrumentation  and  other  resources. 

•  Training  Evaluation:  course,  instructor,  and  trainee  evaluation  including  analysis  and  reporting. 

•  Safeguarding:  making  the  necessary  safety  provisions  for  the  execution  of  EVS,  including 
sufficiently  clearing  the  environment  in  which  ET  exercises  are  carried  out  from  safety  threats  and 
ensuring  safe  transitions  between  ‘training  mode’  and  normal  ‘operational  mode’  and  vice  versa. 

For  the  purposes  of  this  paper,  the  authors  do  not  distinguish  between  the  term  ‘training  management’  and 
the  term  Teaming  management’.  However,  the  term  learning  management  is  more  commonly  associated 
with  the  term  Teaming  management  system’  or  Teaming  content  management  system’.  These  terms  are 
restricted  to  software  applications  which  are  mostly  used  for  a  limited  set  of  management  tasks  related  to  on¬ 
line  training;  e-leaming  programs  and/or  computer  based  training  programs.  Since  this  paper  deals  with 
management  of  EVS  based  learning  and  Embedded  Training  in  a  broader  sense,  the  term  training 
management  will  be  used. 

Cannon-Bowers  (2010)  argues  that  Embedded  Training  (i.e.,  training  enabled  through  EVS)  is  in  fact 
scenario-based,  and  the  supposed  training  value  is  based  on  realistic  scenarios  as  an  effective  means  to 
accelerate  the  development  expertise.  An  important  training  management  task  in  such  training  is  to  record 
trainee  performance  and  use  that  information  to  guide  future  training  exercises.  However,  Cannon-Bowers 
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contends  that  in  the  operational  environments  in  which  ET  is  intended  to  be  used,  this  management  task  is 
often  accomplished  informally,  rendering  subsequent  training  sessions  less  effective  than  they  could  be.  An 
important  requirement  for  training  management  in  EVS  should  therefore  be  the  precise  detection  and 
registration  of  the  trainees’  level  of  expertise  such  that  subsequent  ET  sessions  and  scenarios  can  be  adapted 
to  the  appropriate  level,  yielding  the  highest  training  value.  This  also  highlights  the  requirement  that  trainee 
performance  during  ET  exercises  should  be  carefully  recorded  for  the  purpose  of  selecting  subsequent 
training  activities. 

1.2  Senior  Personnel  Functions  and  Required  Competencies  in  Managing  an  Embedded 
Training  Environment 

Because  the  EVS  is  embedded  in  -  or  appended  to  -  real  equipment  (e.g.,  to  an  aircraft,  a  ship  or  a  vehicle), 
ET  typically  does  not  have  a  human  instructor  in  the  classical  sense.  The  instructor  function  in  virtually- 
based  ET  is  most  often  performed  by  an  embedded  instructor  which  resides  in  software.  In  some  cases  there 
is  no  instructor  function  and  the  trainee  1  must  learn  from  whatever  feedback  the  ET  provides  in  an 
unorganized  manner.  In  those  cases  the  trainees  must  function  as  their  own  instructor.  In  most  cases,  while 
there  is  not  a  human  instructor  in  the  traditional  sense  in  ET,  there  will  need  to  be  a  senior  person  who  can 
perform  some  of  the  training  management  functions  a  traditional  instructor  would  perfonn. 

1.3  Instructor  Function  Competencies 

The  International  Board  of  Standards  for  Training,  Performance  and  Instruction  (IBSTPI)  has  established 
standards  that  are  useful  in  determining  how  senior  personnel  can  best  interact  with  EVS.  They  have 
described  competencies  necessary  for  instructors,  training  managers  and  training  evaluators.  (IBSTPI,  2010). 
A  supervisor  with  warfighters  using  ET  might  have  to  perform  some,  or  all  of  the  following  competencies 
depending  upon  the  sophistication  of  the  EVS  being  used.  In  Table  4-1  to  Table  4-3,  a  sub-set  of  IBSTPI 
competencies  are  listed  that  apply  to  the  training  management  functions  that  will  have  to  be  addressed  in  any 
embedded  training  system,  either  by  the  automatic  system  or  the  supervisor  of  the  warfighter  receiving 
training. 


Table  4-1:  Instructor  Competencies  (IBSTPI,  2010). 


Competency 

Behavior 

Communicate  effectively. 

Prepare  for  instruction.  The  supervisor  might  need  to  show  the  trainee  how  to 
use  the  ET 

Planning  and  Preparation 

Instructional  Methods  and  Strategies 

Stimulate  and  sustain  learner  motivation  and  engagement 

Demonstrate  effective  presentation  skills 

Demonstrate  effective  facilitation  skills 

Demonstrate  effective  questioning  skills 

Provide  clarification  and  feedback 

Promote  retention  of  knowledge  and  skills 

Promote  transfer  of  knowledge  and  skills 

Assessment  and  Evaluation 

Assess  learning  and  performance 

Evaluate  instructional  effectiveness 

Management 

Manage  an  environment  that  fosters  learning  and  performance 

Manage  the  instructional  process  through  the  appropriate  use  of  technology 

1  While  the  term  ‘trainee’  is  used  in  this  section,  it  should  be  construed  to  mean  the  actual  Warfighter  who  is  in  the  field  or 
garrison.  While  these  warfighters  may  not  be  trainees  in  the  formal  schoolhouse  sense,  when  they  are  receiving  embedded 
training  in  the  field  they  are  trainees  for  that  short  period. 
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Table  4-2:  Training  Manager  Competencies  (IBSTPI,  2010). 


Competency 

Behavior 

Communicate  effectively  in  visual,  oral  and 
written  form 

Maintain  networks  to  advocate  for  and  support  the  training  function 

Design  and  Development 

Evaluate  training  and  performance  interventions 

Assessment  and  Evaluation 

Assess  learning  and  performance 

Evaluate  instructional  effectiveness 

Administration 

Apply  leadership  skills  to  the  training  function 

Apply  management  skills  to  the  training  function 

Table  4-3: 

Evaluator  Competencies  (IBSTPI,  2010). 

Competency 

Behavior 

Communicate  effectively  in  visual,  oral  and 
written  form 

Implementing  the  Evaluation  Plan 

Collect  data.  The  supervisor  might  be  called  upon  to  collect  data,  both 
analytical  and  empirical,  for  evaluators  of  the  ET 

Managing  the  Evaluation 

Monitor  the  management  plan 

Work  effectively  with  personnel  and  stakeholders. 

In  addition  to  the  IBSTPI-competencies,  the  supervisor  might  have  to  make  use  of  the  additional 
instructional/management  functions.  These  functions  are  listed  in  Table  4-4. 

Table  4-4:  Additional  Supervisor  Competencies  needed  to  Manage  EVS. 

Competency 


Providing  additional  feedback  from  the  training  session,  via  a  formal  or  informal  After  Action  Review 


Remediation  prescriptions 


Record  keeping 


Planning  the  training  path  forward  for  the  trainee(s) 


Reporting  individual  and  aggregate  progress  up  the  chain  of  command\ 


Determining  if  the  trainee(s)  are  ready  for  operations  on  their  weapon  system 
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While  many  of  these  functions  might  be  performed  by  automated  systems  within  the  EVS,  a  human  may 
need  to  perform  them  for  the  following  reasons: 

•  the  EVS  developers  are  not  able  to  produce  a  system  that  handle  those  functions  automatically, 
and/or 

•  the  unit  is  more  comfortable  having  a  supervisory  human  in  the  loop. 

Even  if  the  EVS  is  completely  automated,  and  the  functions  shown  above  are  done  without  need  of  a  human 
supervisor,  the  supervisor  will  still  need  to  be  informed  of  the  results  of  the  training.  This  would  include 
information  about  the  strong  and  weak  points  of  the  trainee’s  performance  in  the  training.  In  addition, 
remediation  suggestions  to  the  supervisor  would  be  warranted  so  that  the  supervisor  could  construct,  in  some 
cases  with  the  trainee,  a  plan  for  improving  performance. 

ET  developers  should  carefully  consider  the  potential  supervisor  functions  and  competencies  that  their  EVS 
designs  will  require.  Generally,  due  to  supervisor  workload  and  lack  of  instructor  training,  it  might  be  a  goal 
to  automate  as  much  as  possible. 

During  the  workshop  of  the  RTG  current  issues  and  practices  with  EVS  were  examined.  These  included 
policy  and  user  requirements,  human  effectiveness,  human  interaction,  and  learning  technologies.  Some  of 
the  presented  papers  considered  training  management  issues.  Additionally  ‘mind-mapping  exercises’  were 
organized  during  this  workshop.  One  of  the  mind  mapping  themes  was  training  management.  The  facilitators 
asked  the  workshop  participants  to  consider  the  following  questions  as  a  means  of  stimulating  discussion: 

•  What  are  viable  instructional  strategies  for  EVS? 

•  What  categories  of  metrics  can  be  used  to  manage  EVS? 

•  EIow  do  you  manage  training  sessions? 

•  EIow  do  you  envision  tasks  currently  accomplished  by  ‘white-cell’  personnel? 

The  following  sections  attempt  to  provide  answers  to  these  questions. 


2.0  WHAT  ARE  VIABLE  INSTRUCTIONAL  STRATEGIES  FOR  EVS? 

Over  the  last  twenty  years  or  so,  a  substantial  body  of  research  has  been  devoted  to  instructional  strategies  in 
simulated  environments.  This  led  to  a  substantial  advance  in  our  knowledge  about  principles  of  training  in 
simulated  environments.  Relevant  literature  considers  part  and  whole  task  training,  adaptive  training, 
intelligent  tutoring,  task  loading  effects,  specialized  skills  training,  and  generalized  skills  training. 

Clearly,  the  identification  of  a  suitable  instructional  strategy  for  Embedded  Training  is  a  matter  of  applying  a 
methodology  such  as  Instructional  System  Design  or  Training  Analysis  (see  e.g.,  Farmer,  van  Rooij, 
Riemersma,  Joma,  and  Moraal,  1999).  A  related  question  is,  whether  all  theoretically  suitable  instructional 
strategies  are  practically  implementable  in  EVS,  or  is  implementation  constrained  by  technological  or  more 
fundamental  issues?  The  approximately  twenty  research  papers  that  were  presented  at  the  NATO  HFM-169 
workshop  highlighted  a  relatively  small  subset  of  the  endless  range  of  possible  instructional  strategies  as 
currently  considered  for  embedded  training. 

During  the  mind-mapping  exercises  it  was  brought  forward  that  the  simplest  instructional  strategy  seems  to 
just  confront  the  trainee  with  a  scenario  (Scenario-Based  Training)  using  EVS,  i.e.,  involving  some  degree  of 
simulation.  Thus,  constructing  an  appropriate  scenario  is  the  minimum  effort  that  needs  to  be  made  in  terms 
of  devising  an  instructional  strategy.  This  however,  can  be  a  major  effort  in  the  complex  skill  domains  we 
are  looking  at  for  ET  in  the  military. 
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Additional  instructional  strategies  that  were  mentioned  are:  whole-task  training  with  injection  of  virtual 
entities  into  the  training  scenario,  instructional  strategies  based  on  intelligent  tutoring  (replacing  the  human 
instructor),  instruction  strategies  that  are  based  on  remotely  monitoring  of  the  exercise  and  providing 
instructor  feedback  via  radio  (e.g.,  emphasis  training),  adaptive  training,  in  which  parameters  of  the  training 
environment  (level  of  complexity,  speed,  etc.)  will  change  depending  on  progress  of  the  trainee. 

Depending  on  the  nature  of  the  ‘host  equipment’  on  which  the  EVS  is  implemented  (which  may  range  from 
a  handgun  to  a  navy  vessel),  the  instructional  strategy  may  be  based  on  part-task  training  (e.g.,  individual 
training  of  marksmanship)  with  any  combination  of  part-tasks,  with  or  without  the  whole -task  context  or 
may  be  based  on  whole-task  training  in  the  operational  environment  (e.g.,  team  training  of  fire-fighting  on  a 
ship). 

Cannon-Bowers  (2010)  elaborates  on  the  important  consideration  that  ET  is  generally  Scenario-Based. 
Although  learning  may  occur  spontaneously  in  such  a  scenario-based  training  environment,  it  is  important  to 
optimize  learning  by  embedding  instruction  in  the  scenario-based  training.  Instructional  decisions  may 
influence  the  difficulty  of  the  tasks  presented  to  the  trainee,  the  form  and  timing  of  feedback,  the  nature  of 
hints  and  cues  provided  to  trainees  and  the  spacing  of  practice  opportunities.  Some  of  the  instructional 
approaches  were  based  on  intelligent  tutoring. 

In  this  context  of  intelligent  tutoring,  Sottilare,  (2010)  made  a  case  for  machine  perception  of  trainee  affect  to 
aid  learning  and  performance  in  EVS.  The  perception  and  analysis  of  affects,  such  as  moods  or  emotions, 
should  compensate  for  the  lack  of  human  instructors  in  the  EVS  environment.  In  addition,  in  the  context  of 
intelligent  tutoring,  Heuvelink,  van  den  Bosch,  van  Doesburg  and  Harbers  (2010)  present  intelligent  agent- 
supported  training  in  an  on-board  naval  ship  application.  Teams  onboard  ships  are  large  and  training  of  team 
skills  is  performed,  with  many  of  the  team  members  being  intelligent  agents.  Heuvelink  et  al.  introduce  the 
notion  of  a  ‘director  agent’  to  keep  the  behavior  of  all  other  agents  carefully  controlled,  particularly  in  their 
goal-directedness.  A  director  agent  can  be  a  surrogate  for  a  supervisor,  capable  of  diagnosing  task 
performance,  instructing  intelligent  agents  and  steering  the  simulation,  thus  offering  training  tailored  to  the 
needs  of  the  trainee. 

A  current  implementation  of  EVS  in  the  Joint  Strike  Fighter  F-35  fighter  aircraft  (Bills,  Flachsbart,  Kern.,  & 
Olson,  2010)  provides  for  fighter  pilot  tactical  training  in  temporary  reserved  airspace,  without  the  need  of 
having  real  (mock)  threats  in  the  air  or  on  the  ground.  This  implementation  features  scenario-based  training 
in  an  almost  fully  operational  whole-task  context.  The  airborne  enemies,  their  air-air  missiles  and  ground- 
based  surface-to-air  missiles  are  however  virtual  agents  and  can  be  programmed  into  a  variable  scenario. 
Also,  the  missiles  shot  by  the  F-35  are  simulated  and  performance  outcomes  are  based  on  real-time  kill 
assessment.  This  implementation  allows  for  team  training  through  implementation  of  a  special  datalink  that 
connects  the  aircraft  in  a  formation.  The  traditional  role  of  the  expert-instructor  is  limited,  since  the  trainees 
obtain  more  direct  feedback  from  the  EVS. 

It  can  be  helpful  if  ET  developers  structure  Advanced  Organizers  for  the  training.  David  Ausubel  ( 1 960) 
posited  that  it  is  important  to  present  to  learners  a  cognitive  scaffold  of  the  material  to  be  learned  before  they 
begin  learning  the  details.  Ausubel's  advance  organizer  can  best  be  classified  as  a  deductive  method. 
Deductive  methods  or  reasoning  provide  the  rule  to  follow,  then  the  example,  leading  to  the  correct  answer 
or  learning  (Mayer,  2003).  This  is  the  opposite  of  inductive  methods  of  reasoning  that  provide  the  example 
to  follow  and  then  the  rule. 

Advance  organizers  are  also  highly  useful  in  the  process  of  transferring  knowledge.  Because  of  the  deductive 
reasoning,  students  are  able  to  use  the  rule  then  the  example  for  learning  to  occur.  Mayer  writes  in  his  text, 
‘  . . .  the  effects  of  advance  organizers  should  be  most  visible  for  tests  that  involve  creative  problem  solving 
or  transfer  to  new  situations,  because  the  advance  organizer  allows  the  learner  to  organize  the  material  into  a 
familiar  structure’  (Mayer,  2003). 
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Part  of  the  instruction  strategy  could  be  that  while  being  deployed,  e.g.,  in  Afghanistan,  that  trainees  can 
‘reach  back’  to  their  home  country,  e.g.,  a  school  or  any  other  training  resource.  Finally,  cross-training  was 
mentioned,  i.e.,  team  training  in  which  team  members  take  on  other  team  members’  roles  and 
responsibilities,  resulting  in  shared  awareness  of  the  teams  tasks,  roles,  responsibilities  and  capabilities. 

A  more  general  remark  was  that  the  selection  of  EVS  as  a  training  medium  should  not  be  done  before  an 
appropriate  instructional  strategy  had  been  selected  and  that  the  appropriate  instructional  strategy  depends  on 
the  instructional  objectives  and  the  training  program  as  a  whole. 


3.0  WHAT  CATEGORIES  OF  METRICS  CAN  BE  USED  TO  MANAGE  EVS? 

Based  on  the  notion  of  Scenario  Based  Training,  Cannon-Bowers  (2010)  emphasizes  that  generally, 
performance  measurement  is  challenging  in  high  performance  environments  where  situations  are  unfolding 
rapidly  and  significant  behaviors  run  simultaneously,  such  as  in  team  situations.  Also,  establishing  standards 
of  perfonnance,  i.e.,  a  set  of  criteria  that  describe  desired  performance,  is  hardly  feasible  in  uncertain, 
ambiguous  decision  making  situations  where  there  are  many  possible  ways  to  solve  a  problem. 

In  the  mind-mapping  exercise,  distinction  was  made  between:  (1)  outcome  measures  ( how  well  the 
mission/task  goals  have  been  achieved);  and  (2)  process  measures  {how  mission/task  goals  were  or  were  not 
achieved,  e.g.,  frequencies  of  certain  behavior,  communication,  mutual  support  in  teamwork,  etc.).  At  the 
same  time,  because  ET  relates  to  complex  task  performance,  there  is  a  need  for  multiple  measures,  covering 
multiple  aspects  of  behavior  (behavioral  measures,  psycho-physiological  measures).  In  addition,  the  need  for 
analysis  of  performance-  or  learning  curves  was  mentioned,  an  issue  that  also  has  been  considered  under  the 
heading  of  training  management/leaming  management  systems. 

Behavioral  measures  that  were  mentioned  were  eye-movement  measures  (EPOG,  blink  frequencies,  pupil 
diameter).  It  was  realized  that  certain  performance  measures  may  not  be  applicable  to  ET,  for  example  those 
collected  by  a  human  instructor  for  feeding  back  immediately  to  the  trainee,  because  human  instructors  are 
not  always  present  in  an  ET  concept,  which  highlights  the  need  for  Intelligent  Tutoring  Systems.  Finally,  the 
need  was  expressed  for  getting  a  measure  (cues)  for  the  level  of  expertise  of  a  trainee  before  starting  the  ET 
session,  in  order  to  set  the  appropriate  scenario. 

Supervising  personnel  must  have  valid  and  reliable  measurement  from  the  ET  system  in  order  to  make 
proper  decisions  about  learning  diagnosis  and  remediation.  Regardless  of  whether  the  ET  has  an  automatic 
instructional  management  system,  or  that  function  is  performed  by  a  supervisor,  or  the  management  system 
is  a  hybrid  of  the  two,  it  is  necessary  to  have  a  quality  performance  measurement  system  to  provide  proper 
diagnosis  and  remediation. 

To  construct  a  quality  performance  measurement  system  the  designers  must  first  determine  what  the  types  of 
learning  outcomes  are  being  assessed.  Gagne  and  Briggs  (1979)  tell  us  there  are  five  main  learning  domains 
(Table  4-5): 
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Table  4-5:  Five  Main  Learning  Domains  (Gagne  and  Briggs,  1979). 


Learning  domains 

Sub  domain 

1  .Verbal 

2.1ntellectual 

i.  Discriminations 

ii.  Concrete  Concepts 

iii.  Abstract  Concepts 

iv.  Rule  Using 

v.  Problem  Solving 

3.  Psychomotor  domain 

4.Cognitive  strategies  domain 

5.  Attitude  domain 

In  ET  we  can  expect  there  to  be  training  objectives  for  all  of  the  five  learning  domains.  Designers  must  have 
capabilities  within  the  performance  measurement  system  to  assess  these  domains  and  provide  supervisors 
and  automated  parts  of  the  training  management  system  with  clear  learning  assessments,  diagnoses  and 
remediation  prescriptions. 

The  assessments  must  be  both  reliable,  that  is,  repeatedly  consistent,  and  valid.  In  this  case,  validity  refers  to 
measurements  that  mimic  closely  the  Warfighter  performance  in  the  training  system  that  we  would  expect  to 
see  when  the  Warfighter  employs  the  weapon  system  in  the  actual  combat  setting.  The  measures  should  have 
congruence  to  the  measures  used  for  training  in  the  formal  schoolhouse.  In  the  addition  the  measures  should 
have  face  validity  to  the  supervisors  and  Warfighter  leaders  up  the  chain  of  command.  The  leaders  must  feel 
that  the  performance  measurement  system  is  giving  them  an  accurate  picture  of  how  well  prepared  their 
charges  are  to  take  on  various  missions. 

Undoubtedly,  it  is  imperative  for  performance  measurement  to  diagnose  the  causes  of  observed  perfonnance. 
Clearly,  it  is  insufficient  to  say  that  a  trainee  performed  inadequately.  In  order  to  mediate  such  that 
performance  has  the  chance  to  improve,  it  is  essential  to  understand  why  inadequate  performance  was 
shown.  A  certain  inadequacy  may  be  attributable  to  a  lack  of  skill,  such  as  lack  of  team  coordination, 
suggesting  a  certain  type  of  instruction  or  intervention,  while  another  inadequacy  in  performance  may  be 
attributable  to  inadequate  mission  planning,  which  would  suggest  an  entirely  different  intervention  in 
training. 

3,1  Feedback  (e.g.,  After  Action  Review) 

Feedback  is  an  essential  element  in  Embedded  Training  as  it  is  in  all  forms  of  training.  ET  feedback  must  be 
succinct  if  it  is  to  be  useful  in  the  field.  Generally,  the  number  of  feedback  items  should  be  kept  to  a 
minimum  if  trainees  and  supervisors  are  to  make  use  of  them  on  a  regular  basis.  The  number  of  feedback 
items  should  be  kept  to  no  more  than  a  dozen,  less  if  possible.  There  are  at  least  two  methods  for  providing 
training  feedback  to  the  users  of  ET  systems. 

•  From  on-board  ET  systems-  Trainees  and  supervisors  are  shown  feedback  displays  via  the  weapon 
system’s  operational  displays.  Ideally,  the  same  information  can  be  is  shown  through  multiple  on¬ 
board  displays  so  that  supervisors  and  trainees  can  see  the  feedback  at  the  same  time  for  discussion 
purposes. 
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•  From  off-board  systems  -  The  feedback  information  can  be  shown  on  displays  off-board  the 
weapon  system  as  they  are  relayed  from  the  ET  system  via  cable  or  RF  signal.  The  advantage  of  this 
approach  is  that  the  supervisor  can  review  performance  for  an  entire  team  with  the  team  that  has 
trained  on  multiple  weapon  systems  (a  tank  platoon).  In  addition,  the  team  can  review  their 
performance  without  a  supervisor  present. 

Obviously,  a  key  design  decision  is  how  much  the  supervisor  should  be  involved  in  the  feedback  to  the 
trainee.  The  supervisor  interaction  can  range  from  a  completely  automated  system  that  provides  all  the 
feedback  to  the  trainee  and  merely  gives  the  supervisor  a  summation  of  performance,  to  a  system  that 
requires  all  feedback  to  be  given  to  the  supervisor  and  he/she  provides  the  feedback  to  the  trainee. 

Frequency  and  rapidity  of  feedback  is  a  key  issue.  As  is  the  case  with  any  simulation-based  training  system, 
providing  feedback  in  a  timely  fashion  can  greatly  aid  the  training  process.  Flowever,  research  shows  that 
delaying  feedback  for  a  day  for  higher  order  cognitive  skills  like  decision  making  can  result  in  better  learning 
retention.  The  theory  behind  this  phenomenon  is  that  the  trainee  will  have  more  time  to  mentally  rehearse 
their  performance  and  the  feedback  will  help  them  instantiate  lessons  learned  optimally. 

In  Scenario  Based  Training,  feedback  is  a  primary  mechanism  for  imparting  targeted  objectives;  hence,  it 
must  be  carefully  developed  and  implemented  (Cannon-Bowers,  2010). 

4.0  HOW  DO  YOU  MANAGE  TRAINING  SESSIONS? 

During  the  mind-mapping  exercises,  the  notion  that  ET  seems  to  diminish  the  role  of  the  human  instructor 
was  elaborated  upon.  It  was  suggested  that  this  was  simply  because  simulation  hard/software  is  attached  to 
real  systems  hardware/software,  resulting  in  a  training  medium  that  most  often  doesn’t  provide  physical 
space  for  a  human  instructor.  Thus  feedback  from  a  human  instructor  will  often  be  delayed  to  some  point 
after  the  mission  exercise,  e.g.,  during  an  After  Action  Review  (AAR)  or  a  mission  debriefing.  The  dual  role 
of  the  supervisor  was  mentioned,  probably  meaning  that  he  is  both  supervisor  of  the  trainee  (senior  person) 
and  has  an  instructional  role  in  deciding  what  to  do  next. 

An  obvious  remark  is  that  the  instructor  could  be  replaced  by  an  intelligent  tutoring  system.  At  a  meta¬ 
instruction  level,  the  training  sessions  could  be  managed  by  a  Learning  Management  System  (LMS).  How 
would  it  interface  with  the  EVS? 

Functional  requirements  imposed  upon  such  system  are: 

•  record  keeping  of  exposure  to  specific  training  events  (counting); 

•  scheduling  trainee  events  in  order  to  optimize  the  learning  process; 

•  scenario  configuration  management  (which  scenarios  are  updated  and  ready  for  use); 

•  persistent  trainee  profiling  (possibly  throughout  his  career)  on  the  basis  of  performance  measures. 

5.0  HOW  DO  YOU  ENVISION  TASKS  CURRENTLY  ACCOMPLISHED  BY 
‘WHITE-CELL’  PERSONNEL? 

‘White-celF  personnel  are  the  support  personnel  during  live  exercises,  e.g.,  fulfilling  role-playing  tasks, 
exercise  performance  scoring,  refereeing,  and  control  of  opposing  forces  (semi-automated  forces).  Some 
participants  in  the  mind-mapping  sessions  hypothesized  that  ET  implementations  could  make  ‘white-celF 
personnel  superfluous  by  automating  their  tasks.  This  is  certainly  dependent  on  the  level  of  intelligence  and 
autonomy  of  the  various  entities  in  the  ET  virtual  scenario.  The  question  was  raised  whether  exercise 
planning  software  packages  exist,  presumably  to  allocate  tasks  in  ET  scenarios  to  ‘white-celF  personnel,  or 
rather:  Do  scenario  management  systems  for  real  or  simulated  exercises  currently  support  entities  for  ‘white- 
cells’? 
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6.0  DISCUSSION,  CONCLUSIONS  AND  RECOMMENDATIONS 

Embedded  training  presents  the  ET  designer  with  both  opportunities  and  challenges  when  designing  the 
training  management  system.  The  urgency  and  immediacy  of  the  training  is  easily  impressed  upon  the 
trainee  and  supervisor  because  the  actual  weapon  system  is  being  used  as  the  training  medium.  Gaining  and 
keeping  the  learner’s  attention  should  not  be  a  problem.  Elowever,  because  the  training  is  taking  place  in  the 
field,  the  normal  procedures  for  training  and  feedback  we  would  expect  to  see  in  a  formal  schoolhouse  will 
be  significantly  affected.  Capabilities  to  record  and  store  training  data  may  not  be  seen  as  a  priority  and 
therefore  important  learning  trend  data  that  is  captured  in  a  schoolhouse  may  be  lost.  This  loss  can  make  it 
difficult  to  plan  for  training  progress  as  the  automatic  ET  system  or  the  supervisor  plans  the  next  steps. 

It  may  well  be  that  both  trainees  and  their  supervisors  will  need  to  receive  training  in  how  to  manage  the 
training  when  ET  is  the  medium.  This  training  coidd  either  be  provided  before  the  trainees  depart  for  their 
combat  theater,  or  it  might  be  provided  as  part  of  the  ET  on  board  the  weapon  system.  Elowever,  particularly 
if  the  unit  is  going  to  a  combat  zone  the  preparations  for  that  deployment  may  prevent  any  time  being 
devoted  to  this  type  of  training. 

Embedded  training  via  virtual  means  is  steadily  improving  in  quality  and  coming  down  in  cost.  Many  of  the 
past  reasons  that  ET  did  not  have  widespread  success  are  now  being  overcome.  Elowever,  continual  progress 
must  be  made  both  conceptually  and  technically  on  the  training  management  systems  if  ET  is  to  have  large 
scale  success.  We  have  provided  a  few  ideas  about  those  areas  of  improvement  in  this  brief  paper. 
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1.0  INTRODUCTION 

The  purpose  of  embedded  training  (ET)  is  to  make  use  of  operational  equipment  so  that  operators  can  train 
effectively.  The  implementation  of  ET  requires  fundamental  capabilities,  including  the  presentation  of 
scenarios,  operator  inputs,  performance  assessment,  and  feedback  (Morrison  &  Orlansky,  1977).  Embedded 
Virtual  Simulation  (EVS)  is  the  name  given  to  the  technologies  that  allow  an  operator  to  sense  and  interact 
with  the  operational  equipment  and  a  virtual  environment  (VE)  that  is  different  from  the  real  one  in  which 
the  operator  is  physically  present.  The  VE  could  be  a  three-dimensional  computer-generated  environment  or 
it  could  consist  of  both  computer-generated  and  real  components  within  or  outside  the  operational 
equipment.  For  example,  a  tank  gunner  could  look  through  his  real  sight  and  see  computer-generated  targets 
superimposed  on  the  local  landscape.  This  combination  of  real  and  synthetic  imagery  is  called  augmented  or 
mixed  reality. 

The  human  interface  to  EVS  systems  could  stimulate  several  human  senses  at  the  same  time  to  promote  the 
perception  of  immersion  within  the  VE.  This  effect  could  be  achieved  or  enhanced  with  sensors  and  controls 
that  allow  self-directed  movement  within  the  spatial  environment,  gestures  and  natural  real-time  interactions, 
including  spoken  interactions,  with  animated  virtual  characters  or  avatars.  Recent  and  emerging  technologies 
now  provide  opportunities  for  enhancing  the  uses  and  efficacy  of  ET  by  providing  new  ways  of  interfacing 
operators  with  EVS  systems. 

This  chapter  considers  some  issues  of  human  interaction  with  EVS  by  summarizing  papers  that  were 
presented  at  the  workshop  of  the  RTG  and  the  discussions  that  surrounded  them.  It  is  based  on  a  report  that 
was  presented  at  ITEC  2011  (Magee,  Sottilare  &  Roessingh,  2011);  the  proceedings  of  that  conference  are 
not  openly  available,  so  much  of  the  ITEC  paper  has  been  reproduced  here.  This  chapter  also  adds  discussion 
of  the  implications  of  EVS  for  military  readiness. 
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2.0  WORKSHOP  PRESENTATIONS 

2.1  Human-Robot  Interaction 

Neuhoefer,  Kausch  and  Schlick  (2010)  addressed  the  need  for  human-in-the-loop,  immersive  simulation 
technologies  as  a  means  of  accelerating  safety  assessments  in  the  control  of  industrial  robots.  They  noted  that 
optical,  see-through  Head-Mounted  Displays  (HMDs)  provide  an  attractive  solution  since  they  have  a  small 
footprint,  are  less  expensive  than  alternatives  visual  display  systems  (e.g.,  CAVES)  and  are  very  immersive, 
but  that  the  size,  weight,  and  resolution  of  see-through  HMD  are  problems  for  usability.  An  additional 
problem  is  that  it  is  difficult  to  make  use  of  see-through  HMDs  if  virtual  objects  need  to  occlude  real  objects 
in  the  visual  scene.  Development  efforts  to  address  this  problem  were  described;  they  make  use  of 
addressable  focal  planes  for  combining  images  of  real  and  virtual  objects.  The  industrial  application  involved 
handling  and  cleaning  heavy  parts  with  a  blasting  tool  that  shot  pellets.  Neuhoefer,  Kausch  and  Schlick 
determined  that  visual  depth,  haptic,  and  auditory  cues  were  important  for  the  simulation.  They  compared 
two  implementations  of  the  human  interface  with  a  stereoscopic  ITMD.  In  an  augmented  reality  (AR) 
condition,  the  operators  were  able  to  see  their  hand  and  the  real  tool.  In  a  VR  condition,  only  computer¬ 
generated  images  were  provided.  The  performances  of  40  users,  and  the  workloads  that  they  experienced, 
were  measured  for  each  condition.  Significantly  more  virtual  pellets  were  used  to  perform  the  task  with 
augmented  reality.  The  participants  reported  that  more  pellets  were  needed  because  their  aim  was  affected  by 
mismatch  in  the  orientation  of  the  pellets  and  the  blasting  tool.  No  significant  difference  in  any  measures  of 
workload,  using  the  NASA-TLX  questionnaire,  was  found.  In  comparison,  the  participants  preferred  the 
interface  that  used  augmented  reality  as  a  potential  solution  because  they  could  immediately  see  their  hand 
move. 

2.2  Spatial  Perception 

Sandor  et  al  (2010)  also  considered  the  advantages  and  disadvantages  of  HMDs.  They  noted  the  benefit  of 
HMDs  for  conveying  three-dimensional  information  and  the  problems  of  a  reduced  field-of-view  and  a  head- 
fixed,  visual  frame.  They  were  concerned  that  these  characteristics  woidd  negatively  affect  visual 
orientation,  particularly  the  perception  of  the  direction  of  gravity.  They  first  conducted  a  study  to  determine 
if  a  tilted,  large-scale,  immersive,  virtual  environment  would  affect  judgments  of  verticality  in  the  same  way 
as  tilted  real  environments.  Variations  of  the  Rod  and  Frame  Test  (RFT)  were  used.  Three  different  virtual 
environments  were  presented:  one  replicated  the  traditional  RFT,  one  provided  wall  paper  on  the  walls,  and 
another  added  furniture  and  objects  to  enhance  depth  cues.  In  addition,  two  methods  were  used  to  adjust  the 
rod.  One  required  use  of  a  computer  mouse  to  adjust  the  vertical  orientation  of  the  rod;  the  other  used  a  real, 
hand-held  rod.  A  typical  sinusoidal  relationship  between  the  amount  of  frame  tilt  and  rod  tilt  was  found  for 
the  real  and  the  virtual  enviromnents.  Within  the  virtual  environments,  the  amplitude  of  the  sine  wave  grew 
with  scene  detail.  In  turn,  these  functions  were  affected  by  the  adjustment  method.  The  condition  that 
allowed  the  participants  to  hold  the  rod  reduced  misperception  of  the  gravitational  direction.  Hence,  there  is 
evidence  that  haptic  control  and  feedback  reduce  the  misleading  effects  of  greater  scene  detail.  This  evidence 
provides  an  argument  for  multi-modal  cueing  in  ET  applications  involving  spatial  orientation. 

Sandor  et  al  also  reported  the  results  of  a  second  study  that  investigated  the  spatial  relationship  between 
visual  and  auditory  cues.  In  a  dark  room,  a  spot  of  light  and  a  sound  were  presented  at  the  same  time,  at  the 
same  location  or  in  different  locations,  on  a  flat,  frontal  plane.  The  participants  judged  whether  a  light  and 
sound  were  fused,  that  is,  whether  or  not  they  appeared  to  be  coming  from  the  same  position  in  space.  The 
limits  for  fusion  were  found  to  be  greater  in  height  than  in  width  (at  eye  level)  and  progressively  greater  for 
stimuli  as  they  moved  away  from  straight  ahead.  To  determine  if  gaze  direction  affects  fusion,  the 
investigators  required  the  participants  to  make  judgments  when  gaze  and  head  orientation  were  aligned  and 
misaligned.  They  found  that  the  fusion  limits  were  influenced  by  both  gaze  direction  and  head  orientation  in 
an  equal  manner. 
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These  results,  obtained  in  a  darkened  room,  replicate  results  in  a  lighted  room  where  the  other  visual  cues 
for  orientation  are  available,  and  they  show  that  fusion  accuracy  varies  with  the  relative  alignment  of  the 
eyes  and  ears. 

2.3  Dismounted  Soldier  Requirements 

A  third  presentation  in  the  workshop  session  on  human  interfaces  was  made  by  Dyer  (2010)  who  was 
concerned  with  the  impacts  that  embedded  training  coidd  have  on  soldier  systems.  Dyer  noted  that  US  Army 
policy  states  that  embedded  training  should  not  adversely  impact  a  system’s  operational  capability  and  that 
embedded  training  needs  to  provide  systems-related  training  and  feedback.  Dyer  also  noted  that  size,  weight 
and  power  are  critical  considerations  affecting  feasibility  of  embedded  training  for  soldier  systems, 
especially  for  fully  embedded,  go-to-war  capability.  Dyer  described  the  requirements  for  embedded  training 
for  the  Ground  Soldier  System  (GSS)  and  the  investigation  of  alternative  architectures  for  embedded 
training,  which  included  virtual  training  in  a  facility  and  a  “stand-alone”  mode.  The  principal  components  of 
the  GSS  are  a  wearable  computer  and  a  GPS  coupled  to  a  helmet-mounted  display  system.  The  conclusion 
was  that  only  a  few  tasks  were  amenable  to  virtual  simulation  and  that  an  approach  for  identifying  the  tasks 
and  skills  that  are  most  appropriate  for  embedded  training  with  the  GSS  was  needed. 

Dyer  described  a  funneling  approach  for  identifying  the  tasks;  it  first  considered  psychological  dimensions, 
such  as  the  memory  decay  for  the  task,  and  task  characteristics,  for  example,  its  frequency.  A  second  step 
considered  military  criteria  that  consisted  of  questions  focused  on  sustainment,  rather  than  initial  skills 
training.  Dyer  described  some  tasks  to  illustrate  the  use  of  the  questions  to  identify  tasks  appropriate  for 
embedded  training,  but  acknowledged  that  the  process  has  not  been  validated.  Dyer  advanced  the  notion  that 
memory  aids  might  be  a  better  and  more  acceptable  means  of  maintaining  skills  than  virtual  exercises.  Dyer 
provided  a  few  examples  of  the  challenges  that  embedded  virtual  environments  need  to  address  for 
dismounted  soldiers.  One  example  of  the  challenges  is  that  many  cues  and  signals  are  used  by  a  fire  team 
getting  ready  to  clear  a  room.  Touch  and  gesture  are  used  by  the  team  members.  Hearing  is  used  to  locate 
hostiles  and  non-combatants.  The  soldier’s  acceptance  of  EVS  as  a  means  of  training  and  rehearsal  was 
identified  as  a  problem  by  Dyer,  who  noted  that  soldiers  gain  confidence  by  practicing  on  ground  similar  to 
the  operational  setting  and  that  their  preference  is  for  live  training. 


3.0  DISCUSSION 

3.1  In-Flight  Training 

The  presentations  at  the  workshop  helped  to  inform  and  stimulate  the  group  discussions  that  followed  as  the 
participants  considered  the  limitations  and  capabilities  of  EVS  as  an  enabling  technology  for  embedded 
training.  The  discussions  were  also  informed  by  the  preceding  addresses  of  the  keynote  speakers  and  several 
presentations  that  were  also  relevant  to  issues  of  the  human  interface.  Among  these  was  the  keynote  of 
Verhaaf  (2009)  who  provided  a  user’s  perspective  on  the  need  to  maintain  high  proficiency  for  air  combat 
while  deployed.  He  described  successful  exploratory  use  of  EVS  by  the  Royal  Netherlands  Air  Force 
(RNLAF)  Command.  It  was  used  for  training  F-16  combat  skills  as  a  precursor  for  application  to  the  F-35 
Joint  Strike  Fighter.  Simulated  threats  were  fed  to  the  sensor  systems  of  the  aircraft  in  flight,  thus  allowing 
several  pilots  to  engage  virtual  targets  in  tactical  manoeuvres  Beyond  Visual  Range  (BVR). 

The  virtual  world  included  the  threats  and  targets,  their  electronic  signatures,  weapons  and  dynamic 
behaviours,  involving  strategies,  tactics,  manoeuvres  and  counter  measures  corresponding  with  their  real- 
world  roles.  This  achievement  demonstrates  that  it  is  technically  feasible  to  include  BVR  engagements  in 
an  EVS  training  exercise.  However,  simulated  engagements  Within  Visual  Range  (WVR),  that  is  within 
about  10  nautical  miles  (depending  on  visual  conditions)  provide  major  technical  challenges  since  they 
require  the  overlay  of  virtual  objects  upon  the  real  world  as  viewed  by  the  pilot. 
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The  visual  requirements  for  training  target  identification  exceed  the  capabilities  of  state-of-the-art  HMDs; 
pixel  resolution  and  (colour)  contrast  are  now  inadequate  for  visual  target  identification  (V1D).  HMDs  or 
suitable  alternatives  do  for  displaying  realistic,  virtual  opponents  in  an  EVS  do  not  yet  exist  (Roessingh, 
van  Sijll,  &  Johnson,  2003).  Roessingh,  van  Sijll  and  Johnson  identify  12  visual  requirements  for  WVR 
embedded  simulations.  These  include  the  following: 

•  An  image  update  rate  and  a  display  refresh  rate  of  at  least  80  Hz,  but  possibly  much  more  than  80 
Hz. 

•  The  time  delay  of  the  WVR-target  visualisation  system  should  be  less  than  20  milliseconds.  In  other 
words,  the  position  in  the  display  of  simulated  targets  should  not  lag  more  than  20  ms  behind  on 
simulated  target  motions,  actual  aircraft  motions,  and  actual  head  motions. 

•  A  field  size  (Field  of  Regard)  of  the  simulated  scene  in  the  range  of  300  degrees  horizontally  and 
150  degrees  vertically. 

•  A  field  depth  in  the  range  of  10  to  18.5  km. 

•  Scene  management  must  be  based  on  point-of-gaze  measurement.  Specification  of  the  type  of  head- 
and  eye-  movements  that  must  be  measured  would  need  to  be  determined  and  depend  on  other 
requirements. 

•  Occlusion  of  virtual  targets  by  real  world  objects  must  be  managed,  that  is,  the  hiding  of  virtual 
targets  or  part  of  a  virtual  target  behind  real  world  objects  which  are  at  closer  distance  to  the 
observer,  such  as  clouds,  mountains  and  aircraft  structures,  must  be  managed.  For  safety  reasons  the 
operational  community  may  be  concerned  about  the  occlusion  of  a  real  target  by  a  virtual  target, 
since  a  mid-air  collision  could  be  possible. 

•  Shading  and  illumination  of  virtual  targets  by  real  world  objects  (sun  and  clouds)  must  be  managed. 

•  Quickly  varying  luminance  levels,  ranging  from  a  few  feet-Lambert  to  several  hundreds  feet- 
Lambert,  in  a  sufficient  number  of  luminance  steps,  should  be  supported  by  the  display  system. 

•  A  resolution  of  the  foveal  image  of  86  pixels  per  degree  visual  angle  (in  both  horizontal  and  vertical 
direction)  should  be  supported.  However,  a  decreasingly  lower  peripheral  resolution  would  be 
needed. 

•  A  bi-ocular  display  with  hundred  percent  overlap  between  the  field  of  view  of  each  eye  is  required. 
When  targets  come  within  a  range  that  is  less  than  160  meters,  support  of  stereopsis  must  be 
considered.  The  latter  implies  a  binocular  display  with  different  images  for  each  eye  corresponding 
with  retinal  disparity. 

•  The  display  method  should  avoid  potential  conflicts  between  accommodation  and  other  monocular 
depth  cues  if  simulated  targets  are  displayed  at  a  short  observation  distance. 

•  The  mechanical  and  optical  properties  of  a  device  that  enables  projection  of  virtual  targets 
superimposed  on  the  real  world  should  not  compromise  pilot  safety  (e.g.,  in  high-g  manoeuvres,  in 
an  ejection  or  in  a  crash). 

Roessingh,  van  Sijll  and  Johnson  note  that  when  only  a  subset  of  fighter  aircraft  WVR  tasks  need  to  be 
trained  with  ET,  some  of  these  requirements  could  be  relaxed.  Roessingh  and  Verhaaf  (2010)  later  provided 
evidence  of  the  positive  training  effectiveness  of  the  approach  and  discussed  how  EVS  addresses  user  needs. 
They  identified  the  visual  display  of  simulated  entities  within  visual  range  as  a  current,  technological 
limitation  and  thus  conclude  that  EVS  is  now  applicable  only  to  training  scenarios  that  do  not  depend  on 
visual  sight  of  the  threats  or  friendly  forces.  However,  when  compared  to  full  mission  simulation,  EVS  in  the 
air  provides  important  sensory  cues  that  are  expensive,  or  difficult  to  provide  on  the  ground  with  full  fidelity. 
These  include  accurate  physical  motion,  control  loading  (force  reflection)  and  aerodynamic  cues. 
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Since  displays  (e.g.  for  radar  and  radar  warning  receiver)  in  the  cockpit  can  contain  both  real  and  virtual 
information  at  the  same  time,  operators  should  always  be  aware  which  information  is  real  and  which  is 
virtual.  This  helps  them  to  make  the  appropriate  trade-offs  and  decisions  during  the  training  scenario.  It  is 
clearly  not  desirable  to  perform  a  potentially  unsafe  manoeuvre  or  action  in  response  to  a  virtual  entity,  while 
this  may  be  totally  justified  in  an  operational  situation.  A  potential  implementation  for  symbols  on  a  display 
is  to  give  the  virtual  entities  a  dedicated  supplementary  tag.  In  the  fighter  aircraft  EVS  that  was  used  by  the 
RNLAF,  this  was  accomplished  by  attaching  a  small  “v”  to  each  virtual  symbol  on  all  displays  where  they 
could  appear.  Naturally  such  information  should  be  designed  carefully  in  order  to  guarantee  positive  transfer 
of  training.  Another  effective  strategy  in  the  design  of  displays  is  to  give  symbols  related  to  real  entities  a 
higher  display  priority.  This  way,  symbols  related  to  virtual  entities  do  never  obscure  those  related  to  real 
entities.  More  advanced  means  are  also  possible.  Automatic  monitoring  of  aircraft  and  its  interaction  with  the 
real  environment  can  prevent  unsafe  situations.  This  can  be  accomplished  by  the  continuous  evaluation  of  a 
number  of  safety  rules  by  the  EVS  itself.  The  simulation  immediately  stops  when  one  of  the  rules  is  violated, 
that  is,  when  it  detects  an  unsafe  situation.  Naturally  this  should  be  properly  announced  to  the  operators.  As 
an  example,  the  system  can  monitor  that  an  aircraft  remains  in  a  temporary  reserved  airspace  during  the 
training.  It  could  also  automatically  detect  potential  collisions  with  real  entities  or  real  terrain.  In  an  overview 
of  the  use  of  embedded  training  for  the  F-35,  Bills,  Flachsbart,  Kern  &  Olsen  (2010)  emphasized  safety  of 
flight  as  a  consideration  within  and  following  a  TRAIN  mode. 

3.2  Ground  Vehicles 

Schmidt  (2009)  provided  a  keynote  address  that  presented  an  EVS  solution  for  the  Infantry  Fighting  Vehicle 
(IFV),  and  Shiflett  (2009)  provided  a  keynote  that  presented  embedded  training  as  key  performance 
parameter  for  the  Future  Combat  System.  Schmidt  described  the  use  of  head-mounted  displays  with  the  IFV, 
and  Magee  (2009)  cited  the  development  of  special  vision  blocks  by  the  US  Army  Research,  Development 
and  Engineering  Command  (RDECOM)  as  a  development  that  enables  armoured  crew  the  ability  to  observe 
computer-generated  imagery  of  the  external  world.  These  examples  indicate  that  embedded  visual  system 
technologies  are  technically  ready  for  use  with  ground  vehicles,  but  Schmidt  also  cautioned  that  safety 
concerns  would  likely  constrain  the  movement  of  ground  vehicles  and  thus  the  use  and  effectiveness  of  EVS 
if  physical  motion  cues  are  needed  for  human  learning. 

3.3  Dismounted  Soldiers 

The  act  of  embedding  training  simulations  on  dismounted  soldiers  may  be  one  of  the  most  challenging 
operational  environments  in  which  to  implement  EVS.  The  ability  to  stimulate  the  soldier’s  senses  and  to 
respond  to  his  touch  is  not  only  limited  by  the  simulation,  but  also  by  constraints  that  include  power,  weight, 
processing  power  and  communication  bandwidth.  Power  consumption  is  already  a  major  concern  in  the 
world’s  military.  Battery  power  is  not  yet  efficient  enough  to  support  dismounted  soldiers  over  long  periods 
of  time.  Any  proposed  EVS  solutions  would  have  to  be  fully  embedded  and  not  add  to  the  already  heavy 
load  carried  by  dismounted  soldiers.  These  fully  embedded  solutions  will  likely  tax  the  processing  power, 
communications  bandwidth  and  battery  power  of  the  operational  systems  they  are  contained  within. 

Soldiers  in  the  modem  military  move,  shoot  and  communicate,  but  they  also  are  expected  to  act  as  sensors  in 
the  environment  and  to  negotiate  with  indigenous  people.  An  EVS  for  dismounted  soldiers  must  be  able  to 
present  the  visual  world  in  sufficient  fidelity  to  allow  them  to  train  in  an  environment  that  recreates  the 
operational  environment.  Soldiers  need  to  detect  the  movement  of  threats  at  a  distance.  They  need  to  be  able 
to  identify  potential  targets  as  friend  or  foe,  and  to  interact  with  other  avatars  and  non-player  characters  to 
allow  recognition  of  emotions  and  body  language  during  negotiation  tasks. 

In  Schmidt’s  (2009)  keynote  address,  he  presented  an  EVS  solution  for  the  Infantry  Fighting  Vehicle  that 
included  a  dismounted  capability.  This  was  limited  in  that  it  was  tethered  to  the  vehicle  and  it  provided  a 
fully  virtual  training  environment.  This  dismounted  EVS  did  provide  an  effective  power  solution  in  that  it 
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drew  power  from  the  Puma  vehicle.  However,  it  did  not  incorporate  the  live  environment  as  part  of  the  EVS 
which  Dyer  (2010)  has  suggested  is  critical  to  effective  training.  A  more  effective,  but  also  more  complex 
and  costly  alternative  to  all  virtual  or  all  live  is  mixed  reality.  The  near  term  potential  exists  to  implement 
virtual  targets. 

3.4  Observations  and  Opinions 

The  following  general  observations  and  common  opinions  were  reported  by  the  groups: 

•  New  EVS  technologies,  such  as  augmented  reality,  create  opportunities  for  expanding  the  usefulness 
and  range  of  applications  of  embedded  training.  The  applications  that  were  identified  included  air, 
ground  and  sea  systems  involving  vehicles,  command  and  control  centers,  and  dismounted 
combatants. 

•  EVS  can  be  a  cost-effective  alternative  to  more  traditional  simulator-based  training  for  vehicle- 
based  tasks  and  command  centers,  but  is  not  technically  ready  for  training  dismounted  combatants. 

•  Safety  issues  associated  with  vehicle  use  can  constrain  the  application  of  EVS;  there  was  much 
concern  about  the  interplay  between  training  and  operational  modes,  and  the  movement  of 
operational  equipment  in  an  ET  mode. 

•  There  was  a  general  concern  about  the  need  for  realism,  to  allow  soldiers  to  train  as  they  fight. 
Many  human  factors  were  identified,  including  the  following: 

•  Sensory  cue  fidelity. 

•  Multi-modal  interactions. 

•  Sensitivity  to  interactions  between  simulated  and  real  worlds. 

•  Sensitivity  to  interactions  among  multiple  players  in  team  training  applications. 

•  Consequences  for  human  performance  in  the  EVS  and  subsequently  for  training  transfer. 

•  Human  adaptation. 

•  Unwanted  side-effects,  e.g.,  simulator-induced  sickness. 

•  EVS  technologies  were  considered  to  be  the  most  advanced  for  vision,  but  limitations  were 
identified  for  night  vision  simulation  systems,  depth  cueing,  and  peripheral  vision  (i.e.,  wide  field  of 
view).  The  participants  thought  that  flexible,  transparent  displays  may  soon  provide  a  solution  to  the 
problem  of  presenting  computer  generated  targets  within  visual  range  for  in-flight  fighter  training. 

•  For  hearing,  sound  effects  and  three-dimensional  rendering  are  thought  to  be  well  developed,  but 
limitations  exist  for  simulating  the  location  of  a  sound  source  in  three-dimensional  space,  especially 
if  head-phones  are  not  used,  and  wide  variation  of  abilities  among  humans  was  noted. 

•  For  speech  recognition,  there  is  much  concern  about  the  masking  effects  of  the  often  noisy 
environments  of  ET,  although  success  has  been  achieved  (e.g.,  for  the  F-35). 

•  For  motion  cueing,  the  physical  stimuli  can  be  real  and  uncompromised,  e.g.,  as  EVS  was  used  for 
in-flight  training,  or  completely  absent,  e.g.,  as  EVS  was  used  for  the  infantry  fighting  vehicles  or 
tanks  that  do  not  move  because  it  woidd  be  unsafe  to  operate  them  in  a  training  mode.  Consideration 
was  given  to  the  use  of  motion  seats  as  a  means  of  providing  physical  motion  cues  in  vehicles  that 
are  stationary,  but  the  costs  and  bulk  of  these  systems  was  thought  to  be  prohibitive  and  of  doubtful 
utility  since  much  of  the  literature  on  motion  cueing  says  that  it  is  not  necessary  for  effective 
training  transfer,  for  most  tasks.  Tactile  vests  and  belts  are  also  a  possible,  more  affordable 
alternative,  but  their  effectiveness  is  unknown. 
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•  For  haptic  and  vibration  cueing,  like  motion  cueing,  the  physical  stimuli  can  be  real  and 
uncompromised,  as  EVS  was  used  for  the  in-flight  training  demonstration  for  the  F-16,  or  it  can  be 
absent  or  seriously  compromised,  in  the  way  that  EVS  was  implemented  as  a  Virtual  Reality  (VR) 
experience  for  robot  control.  Consideration  was  given  to  the  use  of  force  loaders  in  a  training  mode 
for  vehicles,  but  the  costs  and  bulk  of  these  systems  was  again  thought  to  be  prohibitive. 

•  For  smell,  odors  can  be  simulated,  but  are  difficult  to  remove  from  the  training  environment. 
Emergency  medicine  was  one  area  where  the  user’s  needs  might  justify  the  use  of  a  chemical 
simulator,  but  few  other  tasks  were  thought  to  need  this  type  of  sensory  cue  for  effective  training. 

•  Inter  and  intra  modal  consistency  was  recognized  as  a  challenge.  For  instance,  the  relationship 
between  simulated  images,  provided  to  a  EIMD  with  head-movements,  and  the  real  world  present  a 
challenge  for  head-tracking  technology.  The  sensitivity  of  human  operators  to  conflicting 
information  within  or  between  sensory  systems  was  identified  as  human  factor  that  coidd  lead  to 
unwanted  side-effects,  such  as  simulator-induced  sickness. 

•  Additional  challenges  exist  in  developing  and  interacting  with  intelligent  agents  (e.g.,  virtual 
humans  and  intelligent  tutoring  systems)  within  EVS.  The  inherent,  remoteness  of  EVS  (especially 
in  deployed  settings)  poses  a  challenging  environment  for  instruction  and  feedback  to  trainees.  It  is 
impractical  to  envision  EVS  in  the  future  where  human  instructors/tutors  would  be  available  to 
support  either  individual  or  collective  training  on  the  scale  required.  It  is  more  likely  that  tutors  will 
be  intelligent  agents  with  adaptive  algorithms  to  provide  tailored  training  to  both  individuals  and 
teams  in  remote  locations. 


4.0  CONCLUSION  AND  RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

The  workshop  revealed  several  successful  applications  that  provided  effective  human  interfaces  to  EVS 
systems.  These  included  interfaces  for  robot  control,  fighter  training,  and  combat  ground  vehicles.  In 
comparison,  effective  human  interfaces  for  dismounted  combatants  were  not  revealed.  For  all  types  of 
systems,  concerns  about  weight,  size,  and  safety,  as  well  as  technological  constraints  were  found  to  restrict 
the  design  and  use  of  the  human  interface  to  an  EVS  system.  For  example,  the  cues  of  physical  motion  are 
limited  with  embedded  training  in  a  tank  because  it  could  be  unsafe  if  movements  of  the  vehicle  are 
unrestricted  during  an  ET  exercise.  In  an  ideal  EVS  implementation,  the  user  will  not  be  able  to  tell  the 
difference  between  the  real  and  simulated  environment.  Thus,  it  will  be  unclear  to  the  user  if  a  failure  is  real 
or  simulated.  Unlike  traditional  simulation,  EVS  needs  to  include  a  method  to  remind  the  soldier  about  its 
mode. 

Human  perception  is  multi-modal.  The  psychophysical  study  (Sandor  et  al,  2010)  of  the  visual  perception  of 
gravitational  direction,  its  susceptibility  to  scene  detail  and  haptic  input,  and  the  psychophysical  study 
(Sandor  et  al,  2010)  of  the  fusion  of  light  and  sound  in  space,  and  its  susceptibility  to  head  and  eye 
orientation,  illustrate  the  interplay  within  and  between  the  senses  in  determining  our  perception  of  the 
environment  and  our  place  in  it. 

The  workshop  participants  concluded  that  the  relationship  between  training  and  real  environments,  that  is, 
the  fidelity  of  the  EVS  and  the  consequences  for  training  transfer,  remain  a  concern  due  to  technological 
limitations  and  a  lack  of  behavioural  information  about  the  efficacy  of  EVS. 

Many  weapon  platforms  are  operated  by  teams,  and  platforms  operate  with  other  platforms  in  many 
missions.  Obviously,  team  training  is  an  important  area  of  application  for  EVS.  Human  Interface 
requirements  need  to  allow  coordination  among  team  members  and  platforms.  This  may  require  dedicated 
communication  channels.  The  use  of  EVS  for  team  training  could  promote  unity  in  operational  procedures 
and  doctrines  and  help  train  effective  communication  techniques.  Training  scenarios  could,  inter  alia,  be 
based  on  actual  battlefield  incidents  involving  factors  related  to  teamwork. 
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As  discussed  above,  artificially  intelligent  agents  are  more  likely  to  be  deployed  with  operational  equipment 
to  support  EVS  in  the  future.  It  is  envisioned  that  part  of  the  human  interaction  problem  space  for  making 
intelligent  tutoring  systems  and  virtual  humans  practical  for  EVS  will  include  low  cost,  unobtrusive  methods 
for  sensing  behaviors  (e.g.,  actions,  gestures)  and  physiology  (e.g.,  heart  rate  and  galvanic  skin  response). 
Behaviors  and  physiology  (observable  trainee  states)  will  then  be  used  to  predict  cognitive  states  (e.g. 
unobserved  trainee  states).  Predicted  cognitive  states  that  are  relevant  to  learning  could  include  affective 
variables  like  frustration  and  confusion  or  others  like  attention  and  engagement.  Methods  to  accurately 
predict  these  cognitive  states  will  determine  the  adaptability  of  any  computer-based  tutor  and  either  limit  or 
enhance  the  trainee’s  perception  of  the  tutor’s  persona,  credibility  and  supportiveness.  In  other  words,  the 
tutor’s  effectiveness  (in  terms  of  learning)  is  likely  to  be  limited  by  the  acceptance  of  the  artificially 
intelligent  tutor  and  acceptance  (or  lack  thereof)  will  be  limited  the  tutor’s  ability  to  predict  the  state  of  the 
trainee  at  least  as  effectively  as  a  human  tutor. 

The  costs  of  simulation  have  long  been  hypothesized  to  grow  exponentially  with  fidelity  while  training 
transfer  has  been  hypothesized  to  increase  monotonically  with  fidelity,  but  with  diminishing  returns  (Miller, 
1 954).  This  is  the  problem  facing  decision-makers  who  must  decide  how  much  to  spend  on  a  simulator,  or 
how  much  to  include  in  an  EVS.  The  plots  of  these  relationships  could  be  very  useful  if  they  were  based  on 
actual  data  since  they  could  be  used  to  identify  the  amount  of  fidelity  that  yields  the  most  training  value  for 
cost.  Elowever,  there  is  no  objective  method  for  measuring  the  overall  fidelity  of  a  training  device  or  EVS 
system.  There  are  few  studies  and  many  different  measures  of  training  transfer.  Thus,  there  is  a  need  to 
develop  measures  and  to  determine  their  relationships.  In  addition,  subjective  opinion,  human  adaptation  and 
simulator-induced  sickness  are  outcomes  that  further  complicate  our  understanding  of  the  human  factors 
associated  with  the  design,  use  and  evaluation  of  EVS. 

On  the  basis  of  the  presentations  and  discussions,  a  number  of  the  workshop  participants  concluded  that  EVS 
should  be  considered  as  an  extension  to  traditional  training  methods  and  that  it  is  not  yet  their  replacement. 

Although  the  implications  of  this  view  were  not  explored  further  at  the  workshop,  a  subsequent  meeting  of 
the  RTG  considered  the  impact  that  EVS  could  have  as  an  extension  of  traditional  training  methods.  The 
RTG  considered  the  impact  that  EVS  could  have  on  military  readiness  by  discussing  it  within  the  context  of 
VR  as  a  potentially  disruptive  technology  (Schlick  &  Alexander,  2011).  Recent  studies  within  NATO  are 
directed  toward  the  identification  of  emerged  or  emerging  disruptive  technologies.  VR  can  sometimes 
provide  an  accessible  and  affordable  alternative  to  full  mission  simulators,  which  have  been  the  traditional 
means  of  exploiting  modelling  and  simulation  (M&S)  for  training.  While  a  VR  system  may  not  provide  all 
the  functionality  of  a  traditional  simulator,  it  can  sometimes  provide  a  more  compelling  means  of  achieving 
immersion  within  a  training  environment.  Immersion  is  often  sought  to  promote  situational  awareness  and 
training  transfer.  These  are  also  principal  goals  of  an  EVS  system. 

Although  VR  training  systems  are  generally  less  reliant  on  physical  reproductions  of  the  operational 
equipment  than  traditional  simulators  and  are  much  less  reliant  on  the  use  of  operational  equipment  than 
EVS  (by  definition)  systems,  they  are  sometimes  better  suited  to  specific  training  objectives  than  full  mission 
simulators  because  the  human  interface  of  a  VR  system  can  be  more  easily  customized  to  match  a  particular 
training  objective  and  its  requirements,  for  example,  the  presentation  of  stereoscopic  images  for  making 
accurate  visual  judgements  of  the  distance  to  an  object  or  a  surface.  Hence,  VR  is  well-suited  to  part  task 
training,  mission  planning  and  mission  rehearsal  applications  where  specific,  rather  than  comprehensive, 
learning  objectives  can  be  identified.  Since  many  VR  technologies  could  be  used  to  interface  an  operator 
with  an  EVS  system,  EVS  systems  could  also  provide  a  better  human  interface  to  a  simulated  environment 
than  traditional  simulators,  and  like  VR,  EVS  systems  could  sometimes  be  better  suited  to  part-task  training 
than  full  task  training.  Safety  concerns,  such  as  limits  on  the  motion  of  operational  equipment  when  the  EVS 
systems  is  in  use,  or  technological  limitations,  such  as  the  current  inability  to  display  simulated  targets  within 
visual  range  of  a  fighter  in  the  air,  will  sometimes  limit  EVS  to  part-task  training.  However,  the  use  of  VR  or 
EVS  systems  for  part-task  training  does  not  necessarily  limit  the  impact  that  they  can  have  on  subsequent 
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task  performance  since  part-task  training  can  be  very  effective  and  disruptive  when  the  training  they  provide 
is  put  to  use.  Consider,  for  example,  the  training  needed  by  the  terrorists  of  9/1 1;  they  did  not  need  to  know 
how  to  complete  take-off  or  landing  with  the  aircraft  that  they  used  as  weapons. 

EVS  systems  could  also  afford  significant  benefits  for  mission  planning  and  mission  rehearsal  since  they 
could  allow  last  minute,  just  in  time  training  that  could  take  advantage  of  the  latest  intelligence  data  for 
scenario  development,  presentation  and  human  interaction.  The  use  of  EVS  systems  for  mission  rehearsal 
has  at  least  two  important  benefits  difficult  to  achieve  with  conventional  training  technologies.  One  is  that 
EVS  provides  a  means  to  exploit  immediate,  intelligence  information,  another  potentially  disruptive 
technology,  to  aid  mission  readiness  beyond  what  could  be  achieved  with  traditional  training  technologies, 
which  are  usually  apart  in  space  or  time  from  the  operational  setting.  Another  training  advantage  of  EVS  is 
that  it  provides  a  means  to  minimize  human  memory  decay  by  closing  the  gap  between  learning,  mission 
rehearsal  and  training  transfer  to  the  operational  task. 

In  conclusion,  the  impact  EVS  systems  could  have  on  military  readiness,  as  an  extension  of  existing 
methods,  should  not  be  overlooked.  EVS  is  a  means  to  exploit  other  potentially  disruptive  technologies  and 
provides  a  potential  means  to  gain  a  training  advantage  over  adversaries  without  this  capability.  Advances  of 
the  human  interface  to  EVS  systems  will  grow  this  advantage. 
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1.0  INTRODUCTION 

During  the  NATO  HFM  research  workshop,  a  special  session  was  dedicated  to  learning  techniques.  The 
discussion  following  the  presentation  was  guided  by  the  following  questions: 

•  What  are  the  factors  enabling/limiting  learning  in  EVS? 

•  When  is  the  optimal  time  to  provide  feedback  in  EVS?  On  error,  on  trainee  request,  other? 

•  How  much  human  involvement  is  required  to  support  learning  in  EVS,  and  how  much  can  be 
delegated  to  intelligent  agents? 

•  How  is  feedback  provided  to  trainees  during  embedded  training? 

A  significant  part  of  the  discussion  on  learning  technologies  centered  on  intelligent  agents  for  embedded 
training  and  how  they  might  support  the  constraints  (e.g.,  lack  of  availability  of  a  human  tutor)  inferred  by 
the  questions  above.  Also  discussed  was  how  they  differ  from  conventional  technology-based  training 
environments. 

So  why  might  intelligent  agents  be  important  to  embedded  training?  In  September  2010,  IBM  asserted  that 
’66  [percent]  of  new  products  have  some  kind  of  intelligence  built  in’.  Less  than  a  year  later,  66  percent  is 
likely  a  low  estimate  of  new  products  that  take  advantage  of  Artificial  Intelligence  (AI)  techniques.  The  need 
for  artificial  intelligence  for  the  masses  highlights  the  expectation  for  user-friendly  products  with  ‘on 
demand’  support.  This  expectation  translates  well  to  training  applications  where  human  interaction  and 
support  (e.g.,  tutors)  are  either  limited,  not  available,  or  impractical  (e.g.,  embedded  virtual  simulations). 
IBM’s  statement  also  parallels  a  need  for  new  equipment  training  in  the  military,  where  expectations  are 
growing  for  products  that  use  AI  to  ‘teach’  the  user  about  their  capabilities  and  how  to  exploit  them. 

Intelligent  agents  take  many  forms,  but  the  NATO  group  explored  three  primary  classes  of  intelligent  agents 
for  EVS  applications:  learning  agents,  Non-Player  Characters  (NPCs)  and  process  managers. 

Learning  agents  are  functional  elements  of  computer-based  tutors  and  support  the  trainee’s  learning  process. 
Agent  functions  might  include  the  development/maintenance  of  trainee  and  expert  models,  the  prediction  of 
the  trainee’s  cognitive  state  (e.g.,  affect  or  motivation),  assessment  of  trainee  performance  and  the 
optimization  of  instructional  strategies  (e.g.,  hints,  questions  or  pumps)  employed  by  intelligent  tutors. 
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NPCs  represent  real  people  or  weapon  systems,  including  their  behaviors  and  cognitive  states  (e.g.,  decision¬ 
making  capabilities).  NPCs  can  be  folly  automated  representations  of  friends,  adversaries,  or  neutral 
characters. 

Process  management  agents  could  be  used  to  automate  laborious  processes  including:  the  development  of 
training  content;  management  of  interfaces  and  network  loading;  and  the  collection  and  analysis  of  data 
during  a  distributed  EVS  training  exercise. 

Intelligent-agent  capabilities,  including  the  evaluation  of  the  trainee’s  ‘readiness  to  learn’,  agent  feedback 
delivery  and  frequency,  the  degree  of  automation,  learning  enablers,  and  the  limitations  of  agent 
technologies,  are  reviewed,  along  with  examples  of  successful  implementations  of  intelligent  technologies 
within  a  sample  of  NATO  countries.  Recommendations  for  future  embedded  training  capabilities  driven  by 
intelligent  technologies  are  also  discussed. 


2.0  INTELLIGENT  LEARNING  AGENTS 

As  noted  above,  learning  agents  are  autonomous  functional  elements  of  computer-based  tutors  that  observe 
and  act  upon  an  environment  and  direct  their  activity  towards  achieving  goals  (Russell  &  Norvig,  2003).  In 
computer-based  tutors,  learning  agents  observe  and  act  upon  information  about  the  trainee  and  his/her 
performance  relative  to  an  ideal  trainee  model,  also  known  as  an  expert  model. 

Learning  agents  might  use  behavioral  and  physiological  sensors,  past  performance  and  other  competency 
measures,  demographic  information,  human  observations  and/or  self-reported  data  to  assess  the  trainee’s 
current  ‘readiness  to  learn’,  predict  their  future  performance  or  cognitive  state.  Learning  agents  might  also 
use  trainee  data  to  build  expert  models  or  to  determine  optimal  instructional  strategies  used  by  intelligent 
tutors.  Instructional  strategies  include,  but  are  not  limited  to:  pace  of  instruction,  challenge  level  of 
instruction,  frequency  of  direction  and  support,  questions  and  pumps.  Examples  and  frameworks  for  learning 
agents  were  presented  and  discussed  during  the  2009  workshop. 

Sottilare  (2009  &  2010)  asserted  that  the  lack  of  human  tutors  within  operational  platforms  limits  the 
understanding  of  each  trainee’s  cognitive  state  (e.g.,  emotional  state)  and  the  completeness  of  the  trainee 
model  within  computer-based  tutoring  systems.  Tutor  technology  is  not  sufficiently  mature  to  provide 
accurate,  portable,  affordable,  passive,  and  effective  sensing  and  interpretation  of  the  trainee’s  cognitive  state 
and  limits  the  adaptability  and  effectiveness  of  the  instruction  in  today’s  embedded  training  systems  and 
other  computer-based  tutor-dependent  environments. 

Jensen,  Mosley,  Sanders  and  Sims  (2009)  reviewed  two  case  studies  of  embedded  training  prototypes 
developed  for  the  U.S.  Army  that  employ  structured  training  methods  to  optimize  learning  without  direct 
instructor  involvement.  These  prototypes  include  a  man-wearable  trainer  for  dismounted  operations,  and  a 
robotic  vehicle  control  station  trainer. 


3.0  INTELLIGENT  AGENTS  AS  NPCS 

Heuvelink,  van  den  Bosch,  van  Doesburg,  and  Harbers  (2009)  utilized  intelligent  agents  as  non-player 
characters  in  a  stand-alone  low-cost  desktop  simulation  used  by  a  single  trainee  who  played  the  role  of  the 
Officer  of  the  Watch  (OW)  in  shipboard  fire  fighting  training  scenarios.  The  Chief  of  the  Watch  (CW), 
Machinery  Control  Room  Operator  (MCRO),  Confinement  Team  Leader  (CTL),  and  the  Attack  Team 
Leader  (ATL)  are  all  agent-based  characters.  This  allows  individuals  to  train  in  realistic  and  complex 
environments  in  the  absence  of  other  human  team  members. 
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Bell  &  Short  (2009)  advocated  the  utility  of  speech-interactive  synthetic  teammates  for  training,  mission 
planning  and  rehearsal.  They  identified  the  following  issues  with  human  role-players:  many  training 
exercises  use  trainees  as  training  aids;  human  role-players  introduced  unwanted  variability  into  the  training; 
sometimes  instructors  were  also  trainees  and  this  complicated  performance  assessment;  and  costs  for  human 
role-players  were  recurring  (e.g.,  compensation  and  transport).  Non-player  characters  were  seen  as  a  viable 
option  to  overcome  these  issues.  As  an  example,  they  created  a  set  of  CAS  scenarios,  focusing  on  dialogue 
between  the  pilot  and  a  Joint  Terminal  Attack  Controller  (JTAC),  which  was  played  by  an  intelligent  agent. 

4.0  INTELLIGENT  PROCESS  AGENTS 

As  noted  above,  process  management  agents  could  be  used  to  automate  laborious  processes  associated  with 
training.  There  is  potential  for  intelligent  agents  to  supplant  painstaking  cognitive  task  analyses  used  to 
develop  expert  models  for  computer-based  tutoring  systems  (Williams,  1993).  Advances  in  sensor 
technology  (e.g.,  unobtrusive  physiological  and  behavioral  sensors)  and  machine  learning  techniques  make  it 
possible  to  produce  more  expansive  and  accurate  expert  models  automatically,  but  additional  research  is 
needed  to  standardize  processes  and  improve  the  accuracy  of  these  models. 

5.0  DISCUSSION 

Morrison  &  Orlansky  (1997)  noted  that  ‘provision  of  individualized  instruction  by  embedded  tutors, 
requiring  little  or  no  supervision’  as  a  common  positive  feature  of  embedded  training  systems  evaluated  at 
that  time.  This  is  based  on  the  assessment  of  trainee  performance  as  part  of  a  simplified  trainee  model.  The 
systems  noted  in  this  study  did  not  consider  physiological  sensors,  past  performance  or  other  inputs  to  assess 
the  trainee’s  cognitive  and  affective  states,  and  therefore  were  limited  in  optimizing  instructional  strategies 
(e.g.,  feedback  or  scenario  adaptation).  This  limitation  could  be  seen  as  a  negative  attribute  of  these  systems 
and  should  be  considered  in  the  design  of  new  EVS. 

An  analysis  of  EVS  requirements  by  the  EIFM-165  RTG  highlighted  significant  technical  challenges  in  the 
development  and  deployment  of  EVS  within  operational  platforms.  Among  those  challenges  were 
communication  and  interaction  with  the  trainee  to  support  real-time  feedback  as  well  as  an  after-action 
review  of  their  performance  during  embedded  training  exercises.  This  involves  more  than  just  movement  of 
information  to  and  from  the  trainee,  but  also  includes  intelligent  observation.  In  live  simulations, 
performance  data  is  collected  via  sensors  on  an  instrumented  range  and/or  by  human  observers,  but  the  use 
cases  identified  by  the  RTG  included  deployed  scenarios  in  un-instrumented  areas  and  without  specialized 
observer  personnel. 

The  RTG  evaluated  technology-based  solutions  and  more  specifically  intelligent  agents  and  their  potential 
to:  overcome  the  EVS  lack  of  knowledge  about  the  trainee  state;  and  answer  some  of  the  guiding  questions 
posed  below: 

•  EIow  should  information  (e.g.,  feedback,  instructional  content,  etc.)  be  provided  to  the  trainee  in 
EVS? 

•  When  (and  how  often)  should  feedback  be  provided  in  EVS  and  is  it  different  than  in  conventional 
training  simulations? 

•  What  level  of  human  involvement  is  needed  in  EVS?  Can  it  be  a  fully  automated  or  semi-automated 
process? 

•  What  are  learning  enablers  and  limitations  in  EVS? 

The  following  provides  discussion  of  the  application  of  intelligent  agents  to  EVS  in  aircraft.  Examples  of 
embedded  training  in  the  aviation  domain  are  primarily  onboard  fighters.  The  predominant  type  of  intelligent 
agents  in  aviation  EVS  applications  is  NPCs. 
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The  literature  on  EVS  in  the  aviation  domain  is  generally  mute  on  the  use  of  intelligent  process  agents. 
Process  management  functions,  such  as  starting  and  stopping  the  EVS,  data  logging,  data  analysis  and 
safeguarding  functions  (which  take  care  of  e.g.,  automatically  terminating  the  EVS  when  flight  safety 
requires)  are  generally  not  implemented  as  intelligent  agents.  A  brief  description  of  the  architecture  that  is 
used  for  such  functions  can  be  found  in  Krijn  and  Wedzinga  (2004),  which  deal  with  an  early 
implementation  of  EVS  for  the  F-16  aircraft.  The  latter  implementation  is  of  the  ‘single-ship’  type,  which 
means  that  aircraft  equipped  with  such  EVS  have  no  dedicated  EVS -communication  link  with  other  live 
aircraft.  As  a  consequence,  these  aircraft  can  only  engage  in  an  EVS  scenario  in  which  they  are  the  only  live 
asset,  i.e.,  they  can  only  act  as  a  single  ship,  not  in  a  team. 

This  is  a  significant  restriction  because  nearly  all  tactics  training  of  fighter  aircraft  is  done  in  formations  of  at 
least  two  aircraft.  Lemmers  (2009)  and  Keuning  (2009)  describe  a  multi-ship  implementation  of  EVS  for  F- 
16  aircraft.  Bills,  Flachsbart,  Kern  and  Olson  (2009)  describe  a  multi-ship  implementation  of  EVS  under 
development  for  the  F-35  aircraft.  The  current  review  of  the  literature  in  the  airborne  domain  doesn’t 
mention  the  application  of  aforementioned  Teaming  agents’  either.  No  documents  in  the  public  domain  were 
found  that  revealed  operational  user  requirements  relating  to  specific  functions  in  EVS  that  support  the 
trainees’  learning  process,  let  alone  the  implementation  of  these  functions  in  the  form  of  intelligent  agents. 

Hence,  the  description  of  the  use  of  intelligent  agents  for  EVS  in  aircraft  must  currently  be  limited  to  the  use 
of  Non-Player  Characters.  Current  efforts  seem  to  focus  on  the  development  of  adversary  NPCs  (Harrison,  et 
al.,  2010).  One  obvious  reason  is  that  the  inclusion  of  synthetic  adversaries  in  EVS  is  highly  relevant  to 
tactical  training  of  fighter  pilots  and  is  cost-effective  in  the  sense  that  fewer  live-assets  in  the  adversary  are 
needed.  Also,  synthetic  adversaries  do  not  need  to  be  equipped  with  complex  communication  architecture  as 
would  be  the  case  with  other  relevant  NPCs  in  a  supportive  or  friendly  role.  After  all,  there  is  usually  no 
communication  through  voice  or  datalink  between  fighter  pilots  and  their  advisories.  Communication 
between,  for  example,  a  live  flight  lead  and  a  synthetic  wing  man  would  be  difficult  to  capture  in  an  EVS 
architecture  and  would  require  a  high  quality  and  robust  implementation  for  the  cockpit  environment,  even  in 
the  least  ambitious  scenarios,  as  was  discussed  by  Roessingh  and  Verhaaf  (2009).  Intelligent  synthetic  team 
mates  in  fighter  formations  will  probably  first  be  applied  for  ground-based  virtual  simulations,  before  being 
applied  in  EVS. 

EVS  for  fighter  aircraft  applications  is  a  relatively  recent  development,  with  the  first  concepts  emerging  in 
the  late  nineties  of  the  last  century.  The  first  implementations  concerned  two  types  of  NPCs:  adversary 
aircraft  and  adversary  ground  threats  in  the  form  of  Surface-to-Air-Missiles  (SAMs).  The  adversary  aircraft 
NPCs  reduce  the  need  to  train  against  live  mock  enemies,  or  ‘red  air’  in  jargon.  The  SAM  NPCs  reduce  the 
need  to  train  at  specifically  instrumented  flying  ranges.  Both  types  of  NPCs  are  considered  to  contribute  to 
substantial  cost  savings,  as  is  further  detailed  by  Bills  et  al  (2009). 

The  first  generation  of  adversary  aircraft  NPCs  is  further  characterized  by  a  virtual  range  from  the  EVS 
equipped  ‘ownship’  which  is  beyond  the  visual  range  of  the  pilot.  Those  NPCs  will  never  enter  the  visual 
range  of  the  pilot  during  an  EVS  scenario.  Hence,  the  training  application  of  those  adversary  aircraft  NPCs  is 
limited  to  Beyond  Visual  Range  (BVR)  scenarios.  According  to  Roessingh,  van  Sijll  and  Johnson  (2003)  this 
is  mainly  for  technological  reasons:  to  realistically  visualize  virtual  adversary  aircraft  in  the  cockpit 
environment,  when  these  aircraft  come  Within  Visual  Range,  is  technologically  complex.  As  a  result,  the 
behaviour  of  these  NPCs  will  only  be  observed  directly  via  cockpit  displays  of  on-board  sensors  (radar,  radar 
warning  receiver,  possibly  infra-red  sensors).  Hence,  the  behavior  that  these  NPCs  need  to  demonstrate  only 
needs  to  be  intelligent  insofar  observable  via  cockpit  displays  in  the  EVS  equipped  aircraft. 

However,  even  intelligent  behaviour  BVR  is  far  from  trivial.  Real  enemies  are,  at  least  to  some  extent, 
unpredictable.  They  seek  to  maneuver  themselves  into  a  better  position  as  the  tactical  situation  changes. 
They  react  to  friend  and  foe.  They  are  adaptable.  In  other  words,  they  are  smart.  The  smart  element  in  the 
behavior  of  these  virtual  opponents  involves  a  number  of  factors.  For  instance,  they  should  be  able  to  detect 
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and  identify  targets  to  attack,  but  should  also  be  capable  of  defending  themselves  against  enemy  action.  And 
just  as  the  individual  characters  of  people  may  vary,  the  variety  in  individual  pilots  and  their  personal  style 
and  preferences  should  also  play  a  role.  The  eventual  objective  of  NPCs  into  EVS  is  to  create  training 
scenarios  where  the  threat  is  realistic,  both  in  the  air  (adversarial  aircraft)  and  on  the  ground  (SAMs).  This 
requires  advanced  cognitive  modeling  and  the  modeling  of  domain  expertise.  To  reach  the  level  of  proficient 
behavior,  NPCs  could  be  ‘trained’,  using  machine  learning  algorithms,  to  instill  the  expertise  that  meets  the 
requirements  of  the  scenarios  and  the  live  participants.  Harrison  et  al.  (2010)  propose  genetic  programming 
to  train  NPCs  for  EVS.  However,  the  maturity  level  of  machine  learning  applications  for  EVS  has  not  been 
demonstrated  as  yet. 

6.0  CONCLUSIONS  AND  RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

Five  critical  areas  were  identified  for  additional  intelligent  agent  research:  technology  (tools  and  methods)  to 
assess  of  the  trainee’s  learning  needs  and  tutor  feedback  based  on  those  learning  needs;  methods  for 
automated  scenario  selection  and  adaptation  based  on  trainee’s  competency  and  learning  needs;  methods  for 
automated  instructional  strategy  selection;  methods  to  optimizing  performance  and  retention;  and  automated 
data  collection  and  assessment  for  after-action  reviews. 

Based  on  the  RTG’s  research,  discussions  and  workshop  outputs,  the  following  conclusions  are  offered 
relative  to  trainee  feedback  in  EVS,  the  application  of  intelligent  agents  in  EVS,  their  limitations  and 
recommendations  for  future  research. 

Feedback  (e.g.,  direction,  support)  should  be  presented  when  the  trainee’s  actual  performance  varies 
sufficiently  from  their  expected  performance.  The  measurement  of  expected  performance  is  largely  based  on 
experience  and  previously  demonstrated  competency  in  the  domain  being  trained.  The  determination  of 
current  performance  is  critical  to  adapting  training.  Given  the  isolation  of  the  trainee  in  an  EVS  from  human 
observers,  performance  assessment  will  be  largely  left  to  computer-based  techniques. 

Feedback  frequency  should  be  driven  by  the  trainee’s  competency,  their  levels  of  engagement,  and  their 
motivation.  Accuracy,  interference,  calibration,  and  human  variability  will  likely  complicate  the  assessment 
of  engagement  and  motivation  through  sensor  technologies.  Feedback  should  also  be  task  dependent  in  that 
tasks  involving  high  risk  and  consequence  might  involve  more  frequent  feedback.  Feedback  should  be 
triggered  by  trainee  mistakes  and  trainee  requests  for  help  or  clarification. 

Intelligent  agents  may  enable  training:  in  situations  where  human  tutors  and  human  role  players,  such  as 
mock-enemies,  are  either  unavailable  or  not  practical/cost  effective;  by  reducing  support  costs  via  automated 
tutors,  scenario  directors,  coaches,  Non-Player  Characters  and  simulation  process  management  agents;  by 
increasing  availability  of  training  through  EVS  as  training  goes  with  the  Warfighter  to  the  theater  of 
operation.  As  they  exist  today,  intelligent  agents  have  limited  adaptability  and  ability  to  motivate  trainees, 
limited  interfaces/interaction  within  virtual  simulations  and  limited  capability  to  perceive  and  understand  a 
trainee’s  cognitive  and  affective  states.  This  limits  the  ability  of  computer-based  tutoring  systems  within 
EVS  (or  other  training  environments)  to  optimally  select  instructional  content  and  strategies. 

NPCs  are  viable  as  tutors,  directors  and  coaches  within  EVS  and  other  simulation  environments,  but 
additional  research  is  needed  to  develop  their  interaction  design  (e.g.,  natural  language  understanding  and 
generation)  and  cognitive  modeling. 

Elements  of  trainee  models  within  computer-based  tutoring  systems  may  include  demographics,  input  from 
behavioral  and  physiological  sensors,  self-reported  and  observed  data  and/or  historical  performance  data. 
Physiological  sensors  are  limited  by  human  variation  and  electromagnetic  interference  (EM).  EM  is  a 
concern  in  ground  vehicles  and  naval  platforms  where  EVS  is  likely  to  be  deployed.  Additional  research  is 
needed  to  assess  what  the  essential  elements  of  trainee  models  are  and  how  they  can  be  used  to  determine  the 
trainee’s  cognitive  state  and  optimal  instruction  in  EVS. 
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In  air  applications,  most  notably  EVS  for  fighter  aircraft  such  as  F-16  and  F-35,  NPCs  are  more  often 
employed  as  adversarial  NPCs  (either  ground  threats,  SAMs,  or  air  threats,  adversary  aircraft).  Currently 
application  is  limited  to  BVR  scenarios,  in  which  adversarial  intelligent  agent  behaviors  are  observable  via 
on-board  sensor  displays  (e.g.,  radars)  only. 

Some  research  challenges  include  display  of  Within  Visual  Range  (WVR)  behaviors,  cognitive  modeling  of 
entities  (e.g.,  weapons  platforms),  tactics  modeling  of  entities  and  multi-agent  collaboration/interaction. 
Machine-learning  approaches  (e.g.,  decision  trees,  Bayesian  networks,  genetic  algorithms)  are  growing  tools 
of  choice. 

Intelligent  learning  agents  show  promise  in  spanning  the  gap  between  human  and  computer-based  tutoring 
capabilities  in  one-to-one  tutoring  domains,  but  additional  research  and  development  is  needed  to  make 
agents  practical  for  use  in  fully  autonomous  training  environments  such  as  EVS. 
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This  report  summarizes  the  main  results  and  outcomes  achieved  during  the  period  of  operations  of  NATO 
HFM-165  RTG  on  ‘Enhancing  Human  Performance  through  Embedded  Virtual  Simulation’  and  the 
associated  HFM-169  Research  Workshop  on  ‘Human  Dimensions  in  Embedded  Virtual  Simulation’.  The 
primary  goal  of  the  group  was  to  address  Human  Factor’s  aspects  specific  to  Embedded  Virtual  Simulation 
(EVS)  for  training  applications  in  the  military  domain.  This  involves  the  effective  and  efficient  use  of  virtual 
and  augmented  simulation  technologies  for  integrating  training  functionality  into  operational  equipment. 
RTG-165  explored  the  potential  of  virtual  simulation  technologies  for  Embedded  Training  (ET)  since  many 
enabling  technologies  have  been  developed  within  the  past  few  years  and  many  innovations  are  on  the 
horizon.  The  recommendations  contained  in  this  report  are  provided  to  support  embedded  virtual  simulation 
as  a  tool  to  enhance  deployed  training  and  improve  overall  military  effectiveness. 

Early  on,  RTG  165  reviewed  and  analyzed  the  concept  of  ET  and  the  lessons-leamed  from  previous 
implementations  of  ET  in  military  systems.  Embedded  training  is  a  concept,  which  tightly  integrates  training 
functionality  into  operational  equipment.  It  allows  military  personnel  to  train  and  rehearse  while  deployed. 
ET  allows  skills  to  be  developed,  maintained  and  adapted  with  deployed  military  equipment.  It  allows  a 
greater  coherence  between  the  operational  environment  (climate,  geographic  area,  coalition  forces,  and 
cultural  aspects)  and  the  training.  It  also  allows  a  wider  range  of  training  opportunities  while  systems  are 
employed  in  military  exercises.  This  enables  scheduling  and  adapting  training  anywhere  and  anytime  with  a 
unit’s  own  resources.  It  enhances  accessibility  of  training. 

ET  is  a  concept  that  has  been  actively  considered  as  an  alternative  for  providing  training  on  combat  systems 
for  many  years,  however,  because  of  technology  and  cost  implications  the  main  applications  of  the  ET 
concept  has  been  in  air  defence  and  naval  systems.  A  review  of  ET’s  history  indicates  that  it  is  a  concept  that 
has  been  popular  with  senior  military  decision  makers,  as  noted  by  the  letter  signed  by  the  Vice  Chief  of 
Staff  and  Under  Secretary  of  the  U.S.  Army  in  the  late  80s  addressed  to  weapon  system  program  managers 
telling  them  that  ET  was  to  be  the  first  alternative  for  providing  training  for  new  weapon  systems.  Despite 
this  policy  decision,  very  few  systems  have  implemented  ET  and  those  that  have  failed  to  go  much  beyond 
the  capabilities  of  existing  simulators. 

To  be  effective  in  deployed  settings  ET  has  to  be  able  to  accomplish  the  functions  that  a  simulation  centre 
staff  and  instructional  infrastructure  perform  but  perform  them  on-board  with  the  members  of  the  training 
unit.  This  goes  well  beyond  requirements  for  incorporating  simulation  capabilities  into  combat  systems  since 
it  addresses  how  training  delivery  and  management  should  occur.  One  possibility  would  be  incorporating 
intelligent  tutor  technology  as  means  for  governing  what  training  is  delivered  to  meet  training  needs. 
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The  RTG  adopted  the  definition  of  ET  used  by  the  U.S.  Army  Research  Institute  (Finley  et  al.,  1988)  in  their 
ten  volume  description  of  ET  requirements  and  processes.  It  defines  ET  as  that  training  which  results  from 
features  incorporated  into  the  end  item  of  equipment  to  provide  training  and  practice  using  that  end  item 
equipment.  It  also  differs  between  three  types  of  ET  systems: 

•  Fully  embedded  systems:  ET  system  is  an  integral  part  of  the  operational  system.  It  is  also  a  part  of 
its  system  architecture  and  has  to  be  considered  during  the  design  of  the  operational  system. 

•  Umbilical  systems:  System  can  be  physically  connected  to  the  operational  equipment  when  training 
is  being  conducted  and  later  disconnected  at  the  conclusion  of  the  training. 

•  Appended  (strap-on)  systems:  System  can  be  appended  or  strapped  onto  the  operational  equipment 
when  training  is  being  conducted  and  later  removed.  With  regards  to  the  required  technology  and 
interfaces,  the  term  “embedded  virtual  simulation”  has  been  defined  by  the  RTG  as  follows: 

‘ Embedded  Virtual  Simulation  is  an  enabling  technology  that  provides  an  interface  to  interactive  simulations 
that  reside  within  or  are  appended  to  the  operational  equipment.  It  can  provide  links  to  local  and/or 
geographically  distant  trainees  and  instructional  resources.  It  enables  a  full  range  of  capabilities  for  aiding, 
learning  and  practicing  individual  and  team  knowledge  and  skills.  ’ 

The  RTG  reviewed  the  state  of  the  art  of  ET  supporting  technologies.  The  group  concluded  that  appropriate 
application  of  Virtual  Reality  (VR),  Augmented  Reality  (AR),  and  Intelligent  Agent  (IA)  technologies  could 
provide  the  means  to  effectively  incorporate  EVS  into  ground  and  air  systems  for  training  both  individual 
and  collective  skills.  VR  and  AR  have  both  been  proposed  as  Emerged  and  Emerging  Disruptive 
Technologies  (E2DT)  by  the  NATO  RTO  HFM  Panel.  Relevant  human  factors  issues  associated  with  these 
two  technologies  have  been  extensively  explored  by  various  RTGs  under  different  Panels.  In  the  early  90s 
VR  systems  became  a  new  and  important  subject  for  research  and  development  in  several  NATO  Nations. 
Cooperative  research  efforts  were  initiated  in  1991  and  1992.  As  one  of  the  first,  UFM-028  provided  detailed 
information  about  the  state  of  the  art  (user  interface  design,  displays,  multimodal  dialogue,  etc.)  and 
important  human  factors  issues  (simulator  sickness,  immersion,  etc.).  Another  RTG  of  the  Information 
Systems  and  Technology  (1ST)  Panel,  IST-01 1,  recommended  a  more  active  exchange  of  information  with 
civil/academic  organizations  in  the  field,  and  maintained  a  technology  watch  on  other  innovative 
technologies.  IST-01 1  concluded  that  a  comprehensive  human  model  embedded  in  intelligent  agents  would 
not  be  possible  in  the  medium  or  the  long  term.  They  felt  that  training  developers  and  researchers  should 
look  closely  at  “intelligent”  behaviors  in  VR,  which  can  be  simulated  on  the  basis  of  well-known  rule-based 
cognitive  architectures  in  conjunction  with  kinematic  and  dynamic  models.  Subsequently,  HFM-121 
concluded  that  VR  technology  was  becoming  mature  enough  to  support  specific  applications  in  education 
and  training.  That  RTG  recommended  continuing  research  and  development  efforts  in  that  area. 
Recommendations  of  RTG- 121  narrowed  the  range  of  applications  of  VR/AR  and  the  results  of  this  work 
served  as  the  initial  impetus  to  look  into  the  potential  of  embedded  virtual  simulation. 

AR/VR  allows  trainees  and  human  operators  to  experience  synthetic  environments  that  prepare  them  for  the 
tasks  they  will  need  to  be  able  to  perform  in  the  real  world.  Those  training  in  a  virtual  or  augmented  reality 
environment  should  be  presented  with  the  same  cues  in  the  synthetic  world  that  they  would  experience  in  the 
real  world.  They  should  be  able  to  react  to  complex  synthetic  stimuli  as  if  they  were  real.  In  this  connection, 
AR/VR  provides  a  technological  approach  that  can  provide  a  realistic  training  environment  and  a  natural 
human-system  interface  by  applying  new  interaction  techniques  and  interface  technologies. 

Their  effectiveness  depends  on  human  perception,  cognition  and  motor  response  demands  of  the  tasks  to 
which  they  are  applied.  Although  VR  is  capable  of  accomplishing  real  time  full  and  part  task  training  that 
rely  primarily  on  visual  perception,  research  is  still  needed  to  help  specify  the  displays  and  interfaces 
needed  for  the  other  senses.  For  example,  training  for  tasks  that  rely  on  manual  dexterity  is  limited  due  to 
poor  haptic  feedback  and  interface  devices.  Poor  interfaces  can  lead  to  negative  side  effects  like  physical 
exhaustion  and  simulator  sickness. 
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Current  representations  of  virtual  environments  do  not  include  the  “mud  and  dirt”  experiences  that  soldiers 
experience  in  the  real  world.  Therefore,  AR/VR  technologies  can  contribute  to  effective  training  strategies 
but  they  will  not  totally  replace  other  education  and  training  methods  or  environments,  particularly,  live 
training.  While  tactics  and  maneuvers  can  be  performed  in  VR,  soldiers  will  always  need  to  experience  the 
physical  demands  and  conditions  of  the  real  world.  Virtual  simulation  can  support  the  effectiveness  of  live 
training  with  the  introduction  of  AR  targets  and  virtual  humans.  VR  will  help  prepare  soldiers  for  live 
training  by  broadening  the  spectrum  of  the  situations  the  soldiers  have  encountered  prior  to  training  on  live 
ranges. 

As  noted  above,  effective  ET  is  not  just  dependent  on  displays  and  interface  devices.  To  be  effective,  ET 
systems  also  need  to  include  effective  pedagogy,  training  strategies,  and  record  keeping.  Advances  in  the 
development  of  computer-based  tutoring  systems  and  serious  games  have  prompted  scientists  to  consider 
their  combined  potential  for  providing  effective  ET.  The  development  of  authoring  tools  to  facilitate 
integrated  game-based  tutors  will  significantly  alter  the  utility,  affordability  and  accessibility  of  quality,  self- 
directed  learning  curriculum  that  is  both  engaging  and  effective.  Research  is  ongoing  to  evaluate  game-based 
tutoring  methods  to  accelerate  learning  and  retention.  Maturation  of  tutor  and  game  technology  will  provide 
powerful  tools  for  creating  effective  ET  systems. 

In  order  to  get  the  perspective  of  the  military  user  community  and  the  defence  industry,  RTG-165  conducted 
a  workshop  in  2009.  During  the  workshop,  presentations  addressed  ET  and  EVS  military  requirements, 
operational  constraints,  training  and  mission  support  requirements  including  intelligent  tutoring,  training 
management,  performance  measurement  and  feedback.  Additionally,  relevant  topics  from  the  domain  of 
human-technology  interfaces,  intelligent  agents  and  directions  for  virtual  simulation  in  operational  platforms 
were  discussed.  Subsequent  brainstorming  sessions  captured  the  main  results  from  interactive  discussions 
following  each  session. 

The  user  requirement  presentations  and  discussion  concluded  that  there  are  different  users  of  EVS  systems: 
trainees,  supervisors,  instructors,  and  other  personnel.  They  have  different  tasks  and  requirements  that  must 
be  taken  into  consideration.  The  analysis  of  training  requirements  session  identified  mission  characteristics, 
task  analysis,  analysis  of  trainees  and  target  groups,  and  training  objectives  as  relevant  topics.  User 
requirements  focused  on  realism  of  the  training  environment,  realistic  behaviour  of  synthetic  players,  team 
training  and  general  usability  of  the  EVS  system.  Training  management  discussions  determined  that  training 
sessions  have  to  be  managed,  methods  and  technological  means  for  instruction  and  feedback  have  to  be 
provided,  and  the  overall  performance  has  to  be  evaluated.  Workshop  discussion  of  cost/benefit  analyses 
concluded  that  they  are  required  in  order  to  determine  affordability  of  EVS  solutions. 

The  RTG  surveyed  military  users  and  potential  users  of  EVS  and  ET  capabilities.  Results  of  the  survey 
indicated  that  users  felt  that  the  concept  of  EVS  was  promising,  although  the  survey  showed  that  many 
service  members  were  not  aware  of  the  ET/EVS  concept  or  its  capabilities.  Those  survey  respondents  who 
were  aware  of  ET  thought  it  provided  distinct  benefits.  Clearly  one  conclusion  derived  from  the  survey  is 
that  potential  users  require  education  about  the  benefits  of  ET  and  EVS.  Another  conclusion  is  that  an 
educational  program  should  be  undertaken  through  both  international  and  national  forums.  The  RTG  started 
to  do  so  by  participating  in  1TEC  2011  where  the  results  of  the  workshop  were  presented  to  researchers, 
military  users  and  decision  makers  in  the  field  of  simulation  and  training. 

Training  management  has  been  identified  as  a  core  topic  for  successful  use  of  EVS.  Because  training  takes 
place  in  the  field  and  not  at  a  designated  training  area  or  schoolhouse,  new  procedures  for  training  and 
providing  feedback  will  be  needed.  Recording,  storing  and  keeping  track  of  training  data  are  required  for 
training  that  is  tailored  to  individual  needs.  Training  on  actual  weapon  systems  adds  to  the  urgency  and 
immediacy  of  the  training.  Management  of  ET  will  also  have  to  address  how  training  should  fit  in  with  the 
other  demands  of  deployed  units.  Training  will  have  to  compete  for  priority  with  operations,  maintenance 
time,  and  rest  cycles. 
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The  human  interface  between  the  weapon  system  mediated  training  system  and  the  user  is  a  key  element  of 
an  EVS  system.  ET  and  EVS  systems  must  be  careful  to  ensure  that  users  understand  that  they  are  either  in 
an  operational  or  training  mode.  Physical  motion  cues  may  be  limited  by  safety  concerns  involved  with  the 
movement  or  vehicles  or  their  components  (e.g.,  a  tank  turret)  in  a  confined  space.  Flight  rules  for  aircraft 
must  be  carefully  thought  through  to  ensure  that  safety  is  not  compromised.  Because  of  the  multimodal 
character  of  human  perception  it  is  important  to  stimulate  senses  appropriately  to  enhance  the  training 
effectiveness  of  the  EVS.  Compared  to  other  modalities,  visual  displays  are  well  developed.  But  there  are 
serious  limitations  in  the  capabilities  of  helmet-mounted  displays,  especially  when  used  in  daylight  outdoor 
environments.  Presentation  of  sound  is  possible  and  supported  since  many  military  jobs  require  the  wearing 
of  headphones.  An  ET/EVS  advantage  was  the  potential  positive  effect  by  minimizing  memory  decay  by 
closing  the  time  gap  between  learning,  mission  rehearsal  and  training  transfer. 

With  regard  to  EVS,  which  applies  AR/VR  technology  to  education  and  training,  some  of  the  most  mature 
applications  have  been  in  command  and  control  domains.  This  includes  command  centres  for  missile  defence 
and  shipboard  AEGIS  systems.  In  these  cases  the  EVS  systems  can  be  used  without  restriction  for  weight 
and  space.  Computational  resources  are  also  available.  In  addition,  operational  C2  displays  can  simply  be 
applied  and  there  is  no  need  for  sophisticated  display  and  presentation  technologies  in  order  to  create  a 
virtual  environment.  Because  the  technological  development  in  these  areas  has  led  to  tremendous  advances, 
EVS  becomes  applicable  for  ground  and  air  platforms.  They  are  typified  in  the  development  of  the  U.S. 
Army  Future  Combat  System  (FCS),  the  Puma  infantry  fighting  vehicle  and  the  F-35  fighter,  all  of  which 
explicitly  required  embedded  training  in  their  overall  design  concepts. 

In  comparison  to  these  applications,  embedded  training  systems  for  dismounted  warfighters  remain  a 
difficult  challenge.  This  is  primarily  caused  by  physical  restrictions,  especially  the  extra  load  and  energy 
requirements  added  by  the  training  system.  It  would  also  require  new  technologies  for  augmenting  the  real 
view  with  computer-generated  stimuli.  A  limiting  factor  is  the  insufficient  brightness  of  current  displays  in 
outdoor  settings.  Existing  prototype  systems  use  an  appended  ET  approach  in  order  to  enable  training  at 
special  indoor  locations. 

A  continuing  challenge  of  the  training  community  in  support  of  EVS  and  ET  will  be  to  push  for  the 
integration  of  these  technologies  into  the  design  of  current  and  future  operational  platforms.  To  do  this,  ET 
and  EVS  should  be  integrated  in  the  system  design  cycle  as  early  as  possible.  The  retrofit  of  existing 
platforms  with  ET  systems  might  cause  additional  technical  problems  and  costs.  Retrofits  may  be  difficult  to 
integrate  properly  with  the  operational  platform.  Even  appended  or  umbilical  EVS  systems  require  a 
consideration  within  system  design  for  optimal  performance.  Since  system  acquisition  designers  are  often 
separated  from  simulation  and  training  designers,  a  more  coordinated  approach  for  the  total  system  design 
may  be  needed. 

The  RTG  concluded  that  EVS  has  the  potential  for  maintaining  the  currency  of  training.  As  already  pointed 
out  above,  human-centered  design  and  integration  of  embedded  virtual  simulation  technologies  requires 
thorough  review  and  analysis  at  the  conceptual,  functional,  and  technical  levels.  EVS  covers  a  wide  spectrum 
of  human-system  integration  spanning  from  novel  pedagogical  concepts  to  innovative  techniques  of  human- 
computer  interaction.  ET  and  EVS  are  promising  concepts  with  a  lot  of  benefits  for  enhancing  military 
effectiveness.  EVS  is  currently  maturing.  As  technology  becomes  improves  and  becomes  more  affordable 
the  likelihood  of  ET/EVS  being  implemented  in  ground  and  aviation  systems  will  increase.  The  goal  of 
having  ET  in  the  U.S.  Army’s  FCS  is  an  indication  of  this  trend.  In  other  domains,  e.g.,  dismounted  soldier 
training,  the  technologies  to  provide  virtual  full  mission  embedded  training  are  not  yet  available. 

Finally,  the  RTG  would  like  to  reiterate  that  for  successful  application  of  ET/EVS  to  future  military  systems 
user  requirements  (including  characteristics  of  the  user,  task,  and  environment)  and  training  management 
will  have  to  be  considered  early  in  the  design  process.  While  training  in  both  institutional  and  unit  settings 
today  requires  an  instructor  that  may  not  be  possible  for  ET/EVS  systems.  An  integration  of  an  intelligent 
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tutoring  system  might  be  able  to  successfully  automate  the  instructor’s  role.  Given  the  key  potential  role  of 
intelligent  tutor  technology  in  the  success  of  ET,  the  RTG  feels  that  further  study  should  be  undertaken  to 
determine  the  maturity  and  promise  of  tutor  technology.  EVS  could  become  a  disruptive  technology  by 
providing  NATO  forces  a  training  readiness  advantage  on  the  battlefield,  particularly  if  EVS  systems  can 
exploit  the  advances  now  being  made  in  other  fields,  such  as  secure,  wide  broadband,  wireless  networking 
and  intelligence  gathering. 
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